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FOREWORD 


This  report  presents  a  revision  and  expansion  of  a  portion  of  NAWCWPNS  TM  7484 
Issues  Affecting  the  Integration  and  Coordination  of  Analysis  at  the  Naval  Air  Warfare 
Center  Weapons  Division  of  October  1993.  That  portion  of  TM  7484  this  report  focuses 
upon  concerns  the  customers  of  analysis,  their  expectations  for  analysis  and  the  resources 
required  to  meet  those  expectations.  The  identification  of  customers  of  analysis  and  a 
preliminary  evaluation  of  their  expectations  were  addressed  in  TM  7484  because  an 
understanding  of  these  elements  is  considered  prerequisite  to  defining  the  analysis 
process  and  performing  quality  analysis. 

Analysis  customers,  customer  expectations,  and  the  resources  required  to  meet  those 
expectations  are  featured  here  not  only  because  they  form  a  proper  base  from  which  to 
define  and  assess  the  analysis  process.  They  also  provide  a  base  on  which  program 
managers  (i.e.,  the  customers  of  analysis)  can  define  and  justify  the  products  they  want 
and  analysts  (i.e.,  the  suppliers  of  analysis)  can  justify  the  time  and  expense  they  must 
incur  to  supply  the  analysis  products — and  meet  the  customer’s  expectations.  Thus,  the 
impetus  for  this  report  is  the  belief  that  the  better  analysis  customers  and  suppliers 
understand  their  roles  and  expectations  i.e.,  the  firmer  the  foundation  upon  which  an 
analysis  is  planned  and  executed,  the  better  the  resulting  analysis. 

This  assessment  was  funded  as  a  fiscal  year  1995  NAWCWPNS  Management 
Support  Item  to  provide  guidance  for  those  involved  in  Cost  and  Operational 
Effectiveness  Analyses  (COE As). 
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EXECUTIVE  SUMMARY 


It  seems  reasonable  to  assume  that  the  customer  of  a  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  is  a  program  manager.  Thus  it  seems  reasonable  that  the 
expectations  of  the  program  manager  are  those  that  should  be  met  as  a  COEA  is 
produced.  But  in  an  analysis  as  complex  as  most  COEAs,  it  is  important  to  recognize 
that  there  is  more  than  one  type  of  customer.  Probably  the  most  important  other  customer 
is  the  analyst  who  actually  synthesizes  the  COEA  from  vast  quantities  and  sources  of 
data,  drawing  upon  many  types  of  tools,  processes,  and  people.  Regardless  of  whether  or 
not  the  analyst  is  a  customer  or  a  supplier,  the  ultimate  goal  of  both  the  program  manager 
and  analyst  is  the  same:  to  support  the  acquisition  of  the  best  system  to  do  the  job. 
However,  they  do  have  different  expectations  about  the  COEA. 

The  analyst’s  fundamental  expectation  is  that  any  analysis  will  be  conducted  with  the 
rigor  and  integrity  inherent  in  the  scientific  method.  Many  expectations  the  program 
manager  has  for  a  COEA  comply  with  those  held  by  the  analyst,  but  other  expectations, 
especially  those  stemming  from  the  need  to  meet  budget  and  schedule  constraints,  may 
not. 


Upon  examination  of  the  expectations  of  the  program  manager  and  the  analyst,  the 
conclusion  is  reached  that  the  program  manager’s  expectations  include  the  analyst’s  plus 
those  imposed  by  schedule  and  budget.  The  expectations  for  quality  COEAs  developed 
from  this  review  of  program  manager  and  analyst  needs  are: 

1.  Consistent  (Repeatable,  Methodological  Integrity) 

2.  Accurate  (Accredited  Models  and  Data;  Trained  and  Experienced  Analysts  and 
Administrators) 

3.  Robust,  comprehensive  results 

4.  Documented  results 

5.  Affordable 

6.  Timely 

7.  Accessible  raw  data  and  results 

8.  Appropriate  use  of  analysis  resources 

9.  Simple  in  concept  and  approach  (understandable) 

10.  Minimum  administrative  burden. 

As  these  ten  expectations  are  considered  in  detail,  the  practical  requirements  for 
meeting  the  expectation  are  addressed.  In  particular,  the  hardware  (including  the 
associated  resources  of  facilities  and  ranges),  software,  procedures,  and  personnel 
required  to  meet  the  expectation  are  identified. 
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INTRODUCTION 


This  document  is  a  guide  for  program  managers  (“customers”)  as  they  seek  to  know 
what  to  expect  when  tasked  to  produce  a  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  (or  for  that  matter,  any  kind  of  analysis),  for  their  program.  These  expectations 
are  developed  from  first  principles  and  are  summarized  along  with  the  resources  required 
to  meet  them  in  the  Appendix  in  Table  A-l. 

This  document  is  also  a  guide  for  analysts  to  remind  them  that,  although  they  are 
usually  considered  the  “supplier”  as  opposed  to  the  “customer”  of  analysis,  they 
necessarily  have  expectations  too.  This  guide  provides  customers  and  suppliers  of 
analysis  an  understanding  of  the  importance  of  various  expectations,  the  interrelationship 
between  customer/supplier  expectations,  and  the  cost  of  those  expectations  in  terms  of  the 
resources  required  to  meet  and  exceed  them. 

The  guide  begins  with  a  brief  description  of  customers  and  suppliers  and  an 
acknowledgment  of  their  differences.  The  differences  are  important,  but  the  goal  of 
addressing  the  role  of  customer  and  supplier  is  to  point  out  that  despite  their  differences, 
they  are  obliged  to  work  together  to  meet  each  others  expectations  and  produce  an 
effective  analysis.  In  part,  this  is  because  both  customer  and  supplier  must  respond  to 
expectations  over  which  they  may  have  little  control,  namely  the  expectations  of  the 
bureaucracy  as  expressed  in  the  documents  that  stipulate  the  contents  of  a  COEA. 
Special  attention  is  given  to  the  expectations  of  this  bureaucratic  “customer”  at  the  close 
of  the  “Customer  Expectations”  section. 


THE  ROLES  OF  CUSTOMERS  AND  SUPPLIERS 

The  customers  using  analysis  in  the  form  of  assessments  in  the  weapons  acquisition 
process  include  a  wide  variety  of  officials  at  all  levels  of  the  defense  establishment.  The 
Department  of  Defense  (DOD)  Defense  Acquisition  Guidelines  of  February  1991,  and  in 
particular,  DOD  5000.2  Part  14,  list  many  of  the  administrators  who  use  assessments  to 
decide  how  and  when  weapon  system  proposals  move  through  the  phases  of  the 
acquisition  cycle  (References  1,  2  and  3).  These  administrators  may  be  considered  the 
primary  customers  of  assessments;  they  include  such  officials  as  the  Secretary  of  Defense 
(SECDEF),  and  deputy  and  assistant  secretaries,  the  service  secretaries,  the  Chiefs  of 
Staff  for  the  Air  Force  and  Army  and  their  staffs,  the  Chief  of  Naval  Operations  (CNO) 
and  his  staff  (OPNAV),  and  the  commanders  of  the  systems  commands  and  associated 
Program  Executive  Officers  (PEOs),  program  managers  (PMAs  in  NAVAIR)  and 
Competency  managers  on  the  NAVAIR  TEAM,  e.g.,  Warfare  Analysis  (4.10),  Cost 
Analysis  (4.2),  and  Survivability  (4.1.8). 

Most  officials  involved  in  acquiring  weapon  systems  find  themselves  in  the  role  of 
both  customer  and  supplier.  Their  roles  are  seldom  exclusive  or  “pure.”  For  example, 
the  program  office  considered  to  be  the  customer  by  the  field  activities  performing  a 
given  analysis  may  very  well  be  the  supplier  of  analysis  to  the  office  of  an  assistant 
secretary.  This  role  reversal  is  a  normal  part  of  the  flow  of  responsibility  in  an 
organization  such  as  DOD.  The  role  of  “customer”  is  not  fixed.  This  fact  forces  us  to 
recognize  that  the  identification  of  customer  expectations  involves  an  awareness  of  a 
considerable  range  of  customer  needs  and  preferences. 
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The  breadth  of  the  range  of  customers  ultimately  includes  all  administrators  and 
scientists  (analysts)  involved  in  the  analysis.  This  is  true  even  at  the  level  where 
individual  analysts  are  running  computer  models  to  “produce”  the  raw  analysis.  Here,  the 
analyst  is  both  customer  and  supplier.  Indeed,  the  analyst  is  a  supplier  of  computer  runs, 
but  at  the  same  time  he  or  she  is  the  customer  of  the  input  data  necessary  to  make  those 
computer  runs. 


Tension  Between  Customers  And  Suppliers 

Often  the  program  manager  (administrator)  of  the  weapon  system  moving  through 
the  acquisition  process  is  considered  the  customer  of  any  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  performed  on  the  given  weapon  system.  This  means  the 
COE  A  is  paid  for  by  the  program  manager  and  it  is  the  program  manager’s  expectations 
for  a  quality  COEA  that  must  be  met.  The  analysts  (scientists  and  engineers)  running  the 
models  and  performing  the  wide  range  of  cost,  survivability,  effectiveness  and  mission 
analyses  involved  in  producing  a  COEA  are  usually  considered  the  suppliers  of  the 
COEA.  These  analysts  strive  to  meet  the  expectations  of  their  customer  and  at  the  same 
time  maintain  their  integrity  as  professional  scientists  who  hold  rather  stringent 
expectations  of  their  own  regarding  a  quality  COEA. 

Morse  and  Kimball  in  their  classic  text  Methods  of  Operations  Research  note  the 
potential  for  tension  between  the  suppliers  of  analysis  (“scientists”)  and  the  customers 
(“administrators”)  because  the  analyst  may  have  a  tendency  to  perform  analysis  to  meet 
the  standards  of  the  scientific  method  whereas  the  customer  expects  quick  assessments 
based  on  limited  data.  This  tension  is  probably  inevitable  and  it  is  no  doubt  a  key  reason 
customers  (administrators  and  scientists)  feel  their  expectations  are  not  met.  According 
to  Morse  and  Kimball  (Reference  4,  p  10a) 

Scientist  [analyst]  and  administrator  [program  manager]  perform  different  functions  and  often 
must  take  opposite  points  of  view.  The  scientist  must  always  be  skeptical  and  is  often 
impatient  at  arbitrary  decisions;  the  administrator  must  eventually  make  decisions  which  are 
in  part  arbitrary  and  is  often  impatient  with  skepticism.  It  takes  a  great  deal  of  understanding 
and  mutual  trust  for  the  two  to  work  closely  enough  together  to  realize  to  the  fullest  the 
potentialities  of  the  partnership. 

Promoting  the  “great  deal  of  understanding”  Morse  and  Kimball  mention  here  is  the 
goal  of  this  report.  We  need  to  know  what  the  expectations  of  the  customers  and 
suppliers  of  analysis  are,  and  we  begin  with  the  expectations  of  the  supplier  of  COEAs, 
the  scientist  or  analyst. 
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EXPECTATIONS  FOR  QUALITY  COEAs 


SCIENTIST  EXPECTATIONS  DERIVED  FROM  THE 
SCIENTIFIC  METHOD 

Analysis  is,  in  general,  considered  part  of  the  greater  discipline  of  scientific  research. 
As  such,  it  is  subject  to  the  rules  and  regulations  of  the  “scientific  method,”  so  it  follows 
that  some  of  the  expectations  the  analyst/scientist  has  for  quality  include  meeting  the 
requirements  of  conscientious,  methodical  science. 

One  author  identifies  six  steps  to  follow  in  exercising  the  scientific  method,  namely, 

1 .  Make  some  general  observations  of  a  phenomenon. 

2.  Formulate  a  hypothesis  about  the  phenomenon. 

3.  Develop  a  method  to  test  that  hypothesis. 

4.  Gather  data  to  use  in  the  test. 

5.  Test  the  hypothesis  using  the  data. 

6.  Confirm  or  deny  the  hypothesis  (Reference  5,  p  38). 

This  list  suggests  that  scientific  research  is  a  controlled  process  wherein  tests  are 
designed  and  conducted  and  observations  are  made  to  form  conclusions.  The  validity  of 
the  conclusions  depends  on  the  ability  of  the  scientist  to  repeat  the  observations,  which  in 
turn,  depends  on  some  accounting  scheme,  i.e.,  scientific  method. 

In  practice,  applying  the  scientific  method  to  analysis  projects  involves  conducting 
“good”  experiments  where  the  goodness  of  an  experiment  is  based  on  meeting  objective 
criteria  such  as  logical  design,  repeatability,  documentation,  and  control  of  intervening 
variables.  For  the  scientist/analyst,  meeting  these  criteria  is  not  only  an  expectation 
associated  with  the  practice  of  the  scientific  method,  it  is  an  essential  part  of  the  method. 

Since  meeting  the  scientist/analysf  s  expectations  for  good  experiments  is  so  critical, 
a  more  detailed  list  of  criteria  for  determining  whether  an  experiment  or  analysis  is  good 
is  presented  in  Table  1.  This  list  is  based  on  D.  R.  Cox’s  Planning  of  Experiments 
(Reference  6).  Along  with  the  list  of  criteria  is  a  list  of  the  expectations  that  may  be 
associated  with  those  criteria. 


SCIENTIST  EXPECTATIONS  REGARDING 
MODELS  AND  SIMULATIONS 

Since  models  and  simulations  have  become  such  a  critical  tool  in  the  performance  of 
assessments,  meeting  the  expectations  associated  with  their  development,  selection  and 
use  are  also  critical  to  the  achievement  of  quality  assessments.  This  critical  link  between 
quality  models  and  quality  assessments  is  recognized  in  the  DOD  5000  series  of 
directives  (References  1,  2  and  3).  The  selection  criteria  for  models  suggested  in  that 
series  is  presented  in  Table  2  along  with  the  expectations  met  when  the  given  criteria  are 
met. 
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A  key  guideline  regarding  the  use  of  models  from  the  Defense  Acquisition 
Management  Policies  and  Procedures  deals  with  using  appropriate  models.  According  to 
the  DOD  manual, 

The  complexity,  scope,  and  output  measures  of  mathematical  models  selected  for  the  analysis  should 
be  appropriate  to  the  system  being  evaluated.  For  example,  a  battalion  size  model  need  not  be  run  to 
evaluate  a  new  truck,  and  an  antisubmarine  warfare  campaign  model  is  not  necessary  for  assessing 
the  performance  of  new  carrier  onboard  delivery  systems.  (Reference  2,  p  4-E-3.) 


TABLE  1.  Criteria  for  Performing  “Good”  Experiments  and  the  Expectations  Associated  With 
Them.  (Much  of  this  list  is  a  paraphrase  of  a  list  found  in  Reference  6,  p  13.  Additional 
sources  on  the  scientific  method  and  experimental  design  are  found  in  References  7  and  8.) 


Characteristic 

Expectation 

1 .  Objects  undergoing  analysis  (e.g.,  candidate  attack  aircraft  systems)  should 
be  evaluated  under  the  same  conditions  regardless  of  which  tools  (models) 
and  procedures  are  used  in  the  evaluation.  To  achieve  repeatability  or  to 
enable  independent  analysts  to  conduct  analysis  simultaneously,  the  tools 
and  procedures  must  be  carefully  controlled  and  the  conditions  (threat, 
environment,  scenario,  candidate  system  characteristics)  clearly  specified. 

•  Consistent 
(Repeatability, 
Methodological 
Integrity) 

2.  Random  errors  of  estimation  should  be  suitably  small.  This  requirement  is 
met  by  determining  the  sample  population  that  must  be  measured  to 
achieve  the  statistical  precision  desired  and  sampling  no  more  or  less  than 
is  required.  In  practice,  this  means  that  when  a  stochastic  model  is  used, 
the  number  of  runs  made  to  measure  a  given  effect — such  as  miss 
distance — should  be  determined  by  a  statistical  assessment  of  the  desired 
precision. 

•  Appropriate  Use 
of  Analysis 
Resources 

3.  The  analysis  should  address  as  wide  a  range  of  operational  conditions  as 
possible  so  that  the  conclusions  drawn  will  be  valid  over  a  wide  range  of 
operational  conditions. 

•  Robust  and 
Comprehensive 

4.  The  analysis  approach  should  be  simple  and  direct. 

•  Simple  in 

Concept  and 
Approach 

5.  A  proper  statistical  analysis  of  the  results  should  be  possible  without 
making  artificial  assumptions.  Examples  of  artificial  assumptions  are 
smooth  terrain,  perfect  communications,  a  clear  electromagnetic 
environment,  or  the  use  of  current  tactics. 

•  Accurate 

•  Robust 

6.  The  analysis  is  not  complete  until  it  has  been  carefully  reported. 

•  Documented 
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The  importance  of  using  quality  (accurate,  appropriate)  data  in  whatever  model  is 
used  is  also  clearly  recognized  in  DOD  5000.2M.  This  guide  stipulates: 

It  is  important  to  develop  a  validated  database  for  the  analysis.  The  data  must  be  current, 
accurate,  and  technically  and  operationally  validated  by  engineering  assessments,  technical 
tests,  and  operational  tests.  Additionally,  current  tactical  and  employment  doctrine  must  be 
reflected  in  the  database.  (Reference  3,  p  8-7.) 

Here  the  expectation  to  be  met  is  for  accuracy  based  on  the  use  of  resources  (data)  that 
have  been  validated. 

TABLE  2.  Selection  Criteria  For  Models  and  Simulations  From  DOD  5000,  and  the 
Expectations  Associated  With  Them.  (Reference  3,  p  8-7.) 


Selection  criteria 

Expectation 

1 .  Like  weapon  systems,  models  are  rarely  entirely  “good”  or  “bad.”  They  are 
suitable  or  unsuitable  for  particular  purposes. 

•  Appropriate  Use  of 
Analysis  Resources 

2.  Models  should  help  eliminate  personal  bias  and  preference.  So  be  cautious 
when  using  models  that  include  a  “operator-in-the-loop  ” 

•  Accurate 

•  Appropriate  Use  of 
Analysis  Resources 

•  Robust  and 
Comprehensive 

3.  A  great  number  of  models  already  are  available  in  almost  every  mission  area. 
Consider  them  before  attempting  to  build  new  ones. 

•  Appropriate  Use  of 
Analysis  Resources 

•  Timely  Assessments 

4.  Keep  the  model  simple.  Often  a  simple  mathematical  equation  can  project  the 
performance  you  are  seeking  to  display. 

•  Simple  in  Concept 
and  Approach 

5.  Be  sure  to  test  the  model  to  see  if  it  describes  the  base  case  well.  Generally, 
we  know  more  about  the  base  case,  the  existing  system,  than  we  do  about  the 
alternatives.  If  the  model  does  not  “predict”  what  we  know  the  existing 
system  can  do,  it  is  not  likely  that  its  other  predictions  will  be  sound. 

•  Accurate 

•  Robust  and 
Comprehensive 

6.  Use  several  models.  If  different  models  yield  similar  results,  one  might  gain 
confidence  that  the  estimates  are  reasonable. 

♦  Consistent 
(Repeatability, 
Methodological 
Integrity) 

•  Appropriate  Use  of 
Analysis  Resources 
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TABLE  2.  (Contd.) 

Selection  criteria 

Expectation 

7.  Run  a  “common  sense”  test.  Are  the  results  plausible?  Are  they  within 
reasonable  bounds? 

•  Simple  in  Concept 
And  Approach 

8.  Evaluate  the  quality  of  the  environmental  simulation  and  the  environmental 
limitation  evaluation.  For  systems  using  sensors  with  a  known  vulnerability 
to  adverse  environmental  conditions,  for  instance,  does  the  model  adequately 
incorporate  the  adverse  effects  of  the  environmental  conditions  during  the 
simulation? 

•  Robust  and 
Comprehensive 

In  addition  to  these  guidelines  for  selecting  models  and  data,  the  analyst  and  program 
manager)  have  expectations  regarding  the  way  the  models  and  data  are  used.  In 
particular,  both  analyst  and  program  manager  must  recognize  that  the  primary  goal  of  a 
COEA  is  a  valid,  fair  comparison;  a  comparison  treating  all  candidates  impartially;  a 
comparison  meeting  expectations  for  methodological  integrity  and  consistency.  In  their 
Cost  and  Operational  Effectiveness  (COEA)  Guide  (Reference  9,  p  53),  the  Air  Force 
Material  Command  stresses  this  critical  aspect  of  the  use  of  models  by  noting  that: 

As  much  as  possible,  the  analyst  should  take  advantage  of  what  a  statistician  would  term 
“matched  pairs”  comparison.  For  example,  if  two  similar  aircraft  are  to  be  compared  for  cost 
and  effectiveness  in  a  particular  mission,  both  aircraft  should  be  evaluated  with  the  same  cost 
and  the  same  operational  effectiveness  models.  It  is  tempting,  but  often  misleading,  to  use 
established,  accepted,  validated  databases  peculiar  to  the  individual  competitors,  particularly 
for  off-the-shelf  buys. 

Here  again,  tension  may  arise  between  the  analyst  striving  for  statistical  validity  and 
the  program  manager  wanting  to  take  advantage  of  good  data  that  has  already  been  paid 
for  and  that  is  available  now.  This  brings  us  to  a  closer  examination  of  some  more  of  the 
expectations  the  program  manager  or  administrator  brings  to  the  COEA. 


ADMINISTRATOR  (PROGRAM  MANAGER) 
EXPECTATIONS 


Time  Related  Expectations 

Among  other  things,  administrators  (including  those  whom  we  in  the  weapons 
acquisition  business  call  program  managers)  expect  good  science  to  be  practiced  in  the 
performance  of  the  analysis  they  request.  This  means  that,  in  theory,  the  analyst  who 
conducts  analysis  to  the  standards  established  for  good  science  will  be  meeting  some  of 
the  administrator’s  expectations.  In  practice,  time  limitations  imposed  on  the  analysis 
erode  this  theoretical  compatibility.  Time,  or  more  accurately  the  lack  thereof,  inevitably 
introduces  a  tension  between  the  exhaustive  research  often  implied  by  the  term  “good 
science”  and  the  responsiveness  required  when  addressing  pressing  operational  problems. 
As  Morse  and  Kimball  note: 

An  important  difference  between  operations  research  [e.g.  mission  effectiveness  analysis]  and 
other  scientific  work  is  the  sense  of  urgency  involved.  In  this  field  a  preliminary  analysis 
based  on  incomplete  data  may  often  be  much  more  valuable  than  a  more  thorough  study  using 
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adequate  data,  simply  because  the  crucial  decisions  cannot  wait  on  the  slower  study  but  must 
be  based  on  the  preliminary  analysis.  The  worker  [analyst]  cannot  afford  to  scorn  superficial 
work,  for  wars  (and,  ofttimes,  industry)  do  not  wait  for  exhaustive  study  (although  the 
exhaustive  study  should  be  made  to  back  up  the  preliminary  work).  (Reference  4,  p  10a.) 

It  seems  one  major  new  expectation  the  administrator/program  manager  has  for  the 
accomplishment  of  a  COEA  is  to  produce  the  analysis  in  a  specified  amount  of  time.  The 
time  factor  puts  pressure  on  the  scientist/analyst  to  relax  many,  if  not  all,  the  criteria  for 
good  analysis  presented  above,  thus  reciprocal  tension  arises. 

Related  to  the  time  constraint  is  the  inevitable  constraint  on  funds  available  for 
performing  a  given  COEA.  The  prudent  scientist/analyst  expects  resources  of  all  kinds  to 
be  used  in  “cost-effective”  ways  as  analysis  is  performed,  but  the  demanding  task  of 
policing  a  COEA  budget  falls  to  the  program  manager.  This  means  the  program  manager 
has  a  very  high  expectation  that  the  COEA  will  be  done  within  budget  constraints;  that 
indeed,  it  will  be  affordable. 


Repeatability  Requires  Access 

Another  job  that  falls  more  to  the  administrators  of  COE  As  than  to  the 
scientist/analyst  performing  the  analysis,  is  relating  the  current  COEA  to  previous 
COEAs  of  the  same  system  (or  to  on-going,  as  well  as  past  COEAs,  of  similar  systems). 
The  need  to  make  such  comparisons  and  even  rebuild  former  COEAs  is  addressed  in 
DOD  5000.2M  in  the  section  addressing  cost  analysis  (Reference  3,  p  15-3).  According 
to  the  manual. 

The  purpose  of  the  documentation  of  cost  estimates  is  to  provide  sufficient  information  about 
the  way  the  estimates  were  produced  so  that  Cost  Analysis  Improvement  Group  analysts 
could,  provided  access  to  the  databases  employed,  reproduce  the  estimates.  The  means  by 
which  each  part  of  the  estimate  was  produced  must  be  fiilly  explained.  [Emphasis  added.] 

This  guidance  suggests  first  that  the  program  manager  should  expect  thorough 
documentation  of  the  COEA,  but  the  key  point  to  be  emphasized  here  is  that  other 
administrators  and  analysts  involved  with  the  COEA-in  this  case,  those  in  the  Cost 
Analysis  Improvement  Group-need  access  to  the  databases  employed  in  performing  the 
COEA.  What  this  means  is  that  the  requirement  for  repeatability  addressed  as  an 
expectation  associated  with  the  scientific  method  is  not  simply  a  hollow  academic 
requirement.  COEAs  are  routinely  repeated  and  such  repetition  is  most  effective  when 
the  raw  data  as  well  as  the  final  results  of  previous  efforts  are  quickly  accessible  in  forms 
that  can  be  clearly  understood  and  used. 


Expectations  Stemming  From  COEA  Evaluation  Criteria 

In  addition  to  the  expectations  of  timeliness,  affordability  and  accessibility,  program 
managers  develop  more  expectations  as  they  follow  the  guidelines  in  DOD  5000.2M  for 
reviewing  or  evaluating  the  quality  of  a  COEA.  According  to  this  guide,  “There  is  no 
formal  checklist  for  reviewing  a  cost  and  operational  effectiveness  analysis”  (Reference 
3,  pp.  8-13  and  8-14),  but  the  manual  does  provide  a  list  of  questions  for  evaluating 
COEAs;  a  list  that  reveals  more  of  the  expectations  of  the  program  manager.  The  list  is 
presented  in  Table  3  as  a  key  to  determining  other  expectations  program  managers  may 
have  regarding  quality  COEAs. 
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TABLE  3.  Questions  For  Evaluating  the  Quality  of  COEAs  and  the  Customer  Expectations 
Inferred  From  Them  (Reference  3,  pp.  8-13  -  8-14). 


Evaluation  question  from  DOD  5000.2M  of  1991 

Expectation 

1.  What  are  the  problems,  deficiencies,  and  opportunities  being 
addressed?  Are  these  symptoms  of  more  basic  concerns? 

•  Simple  in  Concept  and 
Approach 
(Understandable) 

2.  Is  the  context  (i.e.,  threat,  scenario,  environment)  consistent  with  the 
Defense  Planning  Guidance?  Has  a  spectrum  of  threats  been 
considered?  Have  Allied  forces  been  considered  in  appropriate  detail? 

•  Consistent 

•  Robust  and 

Comprehensive 

3.  Have  assumptions  and  constraints  been  identified  explicitly?  Are  they 
reasonable?  How  would  changes  in  them  affect  the  results? 

•  Documented 

•  Robust  and 

Comprehensive 

4.  Have  all  reasonable  alternatives  been  considered? 

•  Robust  and 

Comprehensive 

5.  Were  multiple  measures  of  effectiveness  used?  Do  they  relate  to  the 
performance  thresholds  and  objectives  established  for  the  system?  To 
overall  improvements  in  capabilities? 

•  Consistent 

•  Robust  and 

Comprehensive 

•  Appropriate  Use  of 
Analysis  Resources 

6.  Have  all  relevant  costs  been  displayed?  Has  the  Cost  Analysis 
Improvement  Group  reviewed  the  cost  estimates? 

•  Robust  and 

Comprehensive 

•  Accurate 

7.  Are  the  models  clearly  identified?  Are  they  appropriate  to  the  system 
being  evaluated?  Are  the  input  parameters  defined  explicitly  in  the 
documentation?  Can  the  results  be  replicated? 

•  Accurate 

•  Consistent 
(Repeatability) 

•  Documented 

8.  Has  the  database  for  the  cost  and  operational  effectiveness  analysis 
been  validated  through  engineering  analyses  or  tests? 

♦  Accurate 

•  Documented 

9.  Does  the  analysis  present  all  costs  and  measures  of  effectiveness  for 
all  alternatives?  Have  equal-cost  or  equal-effectiveness  alternatives 
been  examined? 

•  Robust  and 

Comprehensive 

•  Accurate 

10.  Are  the  criteria  used  for  assessing  alternatives  identified  explicitly? 
Are  they  meaningful?  Consistent  with  higher  order  objectives? 
Intuitively  acceptable  or,  if  not,  adequately  explained? 

•  Documented 

•  Simple  in  Concept  and 
Approach, 
(Understandable) 

•  Consistent 

11.  Do  the  results  look  reasonable?  Is  it  clear  from  the  analysis  why  the 
effectiveness  measures  came  out  as  they  did? 

•  Simple  in  Concept  and 
Approach 
(Understandable) 

12.  Were  sensitivity  analyses  conducted  showing  how  changes  in 
technical  performance  affect  military  utility,  cost,  and/or  schedule? 
Do  the  results  suggest  reasonable  ranges  or  thresholds  for  performance 
and  cost?" 

•  Robust  and 

Comprehensive 

•  Simple  in  Concept  and 
Approach 
(Understandable) 
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Program  Managers  Expectations  Of  Models  And  Simulations 

In  the  assessment  of  expectations  that  administrators  might  have  for  COEAs,  a  set  of 
“points”  developed  by  the  Director,  Operational  Test  and  Evaluation  (OT&E)  has  been 
reviewed.  This  list  of  points  is  aimed  at  the  use  of  modeling  and  simulation  in  support  of 
OT&E,  and,  as  such,  it  seems  to  be  oriented  more  toward  the  expectations  of  program 
managers  than  analysts.  (Note:  The  expectations  of  scientists/analysts  regarding  models 
and  simulations  are  presented  in  Table  2.)  The  Director,  OT&E,  emphasizes  the  need  for 
confidence.  The  Director’s  points  are  listed  in  Table  4  with  the  customer  expectations 
associated  with  achieving  the  desired  confidence. 


TABLE  4.  Set  of  Points  Provided  by  the  Director,  OT&E,  For  Establishing  Confidence 
in  Models  and  Simulations.  Also  listed  are  the  customer  expectations  associated  with 
achieving  the  desired  confidence.  (Reference  10,  pp.  6  and  7.) 


Point 

Expectation 

Confidence  in  the  model,  based  on  a  sound,  coherent,  systematic  process  used 
in  development;  sound  model  management  structure,  including  configuration 
management;  model  descriptions,  including  usage,  strengths  and  weaknesses, 
past  history,  adequate  documentation;  thorough  description  of  verification 
and  validation  efforts;  threat  representation  and  usage;  and  thorough 
descriptions  of  accreditation  efforts. 

•  Consistent 

(Repeatability, 

Methodological 

Integrity) 

•  Accurate 

•  Robust  and 
Comprehensive 

•  Documented 

Confidence  in  the  M/S  [Modeling  and  Simulation]  team.  M/S  practitioners 
must  be  experienced  with  the  simulation  being  used  and  the  system  being 
simulated.  That  part  of  the  team  tasked  with  establishing  the  confidence  in 
the  model  must  be  independent  of  the  developers  and  users  of  the  model  or 
simulation. 

•  Accurate 

•  Consistent 
(Repeatability, 
Methodological 
Integrity) 

Confidence  in  M/S  methodology  and  use,  based  on  applicability  or 
appropriateness  to  address  requirements  and  issues  under  consideration, 
adequately  described  methodology  and  assumptions,  certified  and 
documented  input  data  (including  scenarios). 

•  Simple  in  Concept  and 
Approach 

•  Documented 

Confidence  in  M/S  results:  M/S  results  should  be  consistent  with  actual  field 
test  results  when  the  M/S  input  data  are  representative  of  actual  field  test 
conditions.  Further,  M/S  results  should  be  consistent  with  other  M/S  results 
when  their  input  data  are  comparable.  Any  M/S  results  that  appear  counter¬ 
intuitive  must  be  fully  investigated  in  order  to  determine  whether  or  not  the 
results  are  in  error  or  if  the  results  actually  reflect  some  new  insight  not 
previously  anticipated.  M/S  results  must  include  a  description  of  related 
OT&E  field  tests,  apparent  inconsistencies,  and  any  available  resolution  of 
issues  identified. 

•  Accurate 

•  Robust  and 
Comprehensive 

•  Documented 

•  Consistent 
(Repeatability, 
Methodological 
Integrity) 

Confidence  in  M/S  results  can  be  enhanced  by  comparison  to  other  data,  e.g., 
actual  test  results,  other  models,  or  historical  data. 

•  Accessible 

•  Documented 

•  Consistent 

(Repeatability, 

Methodological 

Integrity) 
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Bureaucratic  Expectations 

Bureaucratic  expectations  are  a  special  set  of  expectations  usually  presented  as 
written  requirements  to  which  the  program  manager  must  respond;  they  are  “imposed”  or 
“specified”  expectations.  Perhaps  they  are  more  accurately  categorized  as  the 
expectations  of  the  decision  makers  at  the  Acquisition  Review  Board  level. 

Examples  of  these  requirements  begin  with  expectations  of  the  purpose  for  the 
COEA.  According  to  Reference  2,  p  4-E-l  and  4-E-2,  the  COEA  is  performed  for  three 
reasons.  These  are: 

“1.  .  .  .  by  illuminating  the  relative  advantages  and  disadvantages  of  the  alternatives  being 

considered  and  showing  the  sensitivity  of  each  alternative  to  possible  changes  in  key 
assumptions  (e.g.,  the  threat)  or  variables  (e.g.,  selected  performance  capabilities). 
Accordingly,  the  analysis  takes  the  form  of  a  problem  of  choice.  The  cost  and  operational 
effectiveness  analysis  should  aid  decision  makers  in  judging  whether  or  not  any  of  the 
proposed  alternatives  to  the  current  program  (i.e.,  the  status  quo)  offer  sufficient  military 
benefit  to  be  worth  the  cost. 

2.  ...  by  early  identification  and  discussion  of  reasonable  alternatives  among  decision 
makers  and  staffs  at  all  levels. 

3.  ...  by  providing  the  analytical  underpinning  or  rational  for  decisions  on  a  program. 
Accordingly,  the  analysis  shall  provide  a  historical  record  of  the  alternatives  considered  at 
each  milestone  decision  point.” 

These  goals  are  directly  linked  to  the  decision  maker’s  expectations  for  accurate, 
robust,  comprehensive,  well  documented  analysis  that  is  simple  enough  in  concept  and 
approach  to  be  used  to  communicate  the  judgments  made  in  selecting  a  particular  weapon 
system. 

Additionally,  as  Reference  2  points  out, 

“A  cost  and  operational  effectiveness  analysis  will  typically  draw  on  several  sub-analyses. 

These  include  analysis  of  mission  needs,  the  threat  and  U.S.  capabilities,  the  interrelationship 
of  systems,  the  contribution  of  multirole  systems,  measures  of  effectiveness,  costs,  and  cost 
effectiveness  comparisons.” 

Here  the  point  is  made  that  the  decision  maker  expects  the  COEA  to  be  consistent 
throughout  the  process  of  coordinating  and  integrating  various  types  of  analysis. 
Furthermore,  the  decision  maker  expects  the  analysis  to  be  robust  and  comprehensive 
enough  to  justify  the  decision  in  the  broadest  context  of  decision  alternatives. 

This  foundation  of  expectations  is  further  refined  and  expanded  in  Reference  2, 
where  ultimately,  a  very  detailed  list  of  specified  expectations  is  presented.  A  summary 
of  these  expectations  is  presented  in  Table  5  along  with  the  general  expectations  implied 
in  meeting  the  specified  expectation. 
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TABLE  5.  Specified  Expectations  Delineated  in  the  Acquisition  Guidelines  for  Performing 
COEAs  (DOD  5000.2M  Reference  2)  And  The  General  Expectations  Implied  in  Meeting  Them. 


Specified  Expectation  (Page  Numbers  Are  From 

DOD  5000.2M,  Reference  2) 

General  Expectation 

A  comprehensive  test  and  evaluation  program  is  an  integral  factor  in 
analyzing  operational  effectiveness ...  [p  8-1] 

•  Robust  and  Comprehensive 

...establish  the  level  at  which  the  main  analysis  will  be  performed.... 
Whatever  level  is  chosen,  a  check  should  be  made  of  the  implications 
for  outcomes  expected  at  the  next  higher  organizational  level.  In 
many  cases  it  will  be  necessary  to  consider  results  at  several  higher 
levels.  [p8-2] 

•  Robust  and  Comprehensive 

•  Appropriate  Use  Of  Analysis 
Resources 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

The  scenarios  should  include  a  set  based  on  situations  that  conform 
to  the  scenarios  in  the  Defense  Planning  Guidance....  The  time 
period  selected  for  the  study  should  be  of  sufficient  length  to  measure 
effects  on  mission  capabilities  once  a  system  has  been  deployed  in 
significant  number [s].  [p  8-2] 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

•  Robust  and  Comprehensive 

The  threat  should  be  analyzed  in  sufficient  detail  to  identify ,  with  a 
reasonable  degree  of  assurance,  the  conditions  that  might  exist  when 
employing  the  new  U.S.  system....  Develop  a  range  of  plausible 
threats,  to  allow  for  the  uncertainty  inherent  in  threat  projections. 

[p  8-3] 

•  Accurate 

•  Robust  and  Comprehensive 

Explore  the  implications  of  constraints  on  the  threat  Ensure  that  a 
worldwide  allocation  of  key  threat  forces  underlies  the  specification 
of  threat  elements  that  would  be  engaged  by  the  U.S.  system. 

Consider  logistics,  personnel,  and  infrastructure  factors  that  might 
affect  the  nominal  performance  of  enemy  weapons  systems,  [p  8-3] 

•  Accurate 

•  Robust  and  Comprehensive 

Evaluate  explicitly  the  potential  contribution  of  Allied  forces. 
Describe  Allied  concepts  of  operation,  projected  force  structures,  and 
capabilities.  [p8-3] 

•  Accurate 

•  Robust  and  Comprehensive 

Evaluate  terrain,  weather,  ocean  or  other  pertinent  environmental 
parameters.  For  atmospheric  conditions,  the  analysis  should  be 
supported  by  meteorological  data  describing  both  normal  and 
reasonably  expected  adverse  weather  conditions  under  which  the 
system  would  be  expected  to  operate,  [pp  8-3,  8-4] 

•  Accurate 

•  Robust  and  Comprehensive 

For  each  threat  environment  there  should  also  be  expected  mission 
capabilities.  [p8-4] 

•  Robust  and  Comprehensive 

A  good  analysis  embraces  a  solid  statement  and  analysis  of  the 
organizational  and  operational  plan  for  each  alternative.  These  plans 
describe  the  way  in  which  forces  and  equipment  would  be  arranged 
and  employed  in  battle.  They  address  both  doctrine  and  tactics  in 
explaining  how  a  system  would  be  used  to  accomplish  national 
objectives.  [p8-4] 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

•  Robust  and  Comprehensive 

When  structuring  the  set  of  alternatives,  consider  both  current 
systems  and  improved  versions,  along  with  systems  in  development 
by  the  other  Services  or  Allies  and  conceptual  systems  not  yet  on  the 
drawing  board.  [p8-5] 

•  Robust  and  Comprehensive 

13 


NAWCWPNS  AdPub  080 


TABLE  5.  (Contd.) 


Specified  Expectation  (Page  Numbers  Are  From 

DOD  5000.2M,  Reference  2) 

General  Expectation 

Where  possible,  include  alternatives  in  which  doctrine  and  tactics, 
rather  than  just  hardware,  are  varied,  since  organizational  and 
operational  plans  can  change.  Consider  including  alternatives  with 
potential  to  mitigate  significant,  environmentally  driven,  performance 
limitations,  [p  8-5] 

•  Robust  and  Comprehensive 

It  is  important  to  develop  a  validated  database  for  the  analysis.  The 
data  must  be  current,  accurate,  and  technically  and  operationally 
validated  by  engineering  assessments,  technical  tests,  and  operational 
tests.  Additionally,  current  tactical  and  employment  doctrine  must  be 
reflected  in  the  database,  [p  8-7] 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

•  Accurate  (Accredited  Models  and 
Data;  Trained  and  Experienced 
Analysts  and  Administrators) 

•  Documented 

Cost  estimates  are  as  important  as  operational  effectiveness  measures 
in  the  analysis  Decision  makers  must  combine  cost  considerations 
with  assessments  of  operational  effectiveness  and  potential 
constraints  (e.g.,  timeliness,  political  considerations)  in  weighing 
alternatives,  [p  8-8] 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

•  Accurate  (Accredited  Models  and 
Data;  Trained  and  Experienced 
Analysts  and  Administrators) 

The  analysis  must  address  the  system  quantity  for  which  a  decision  is 
being  sought  The  acquisition  objective,  if  different  can  be  treated  as 
an  excursion.  Quantity  ranges  are  acceptable  if  the  planned  buy  is 
within  the  specified  range,  is  specifically  addressed,  and  assumes  a 
reasonable  procurement  schedule,  [p  8-8] 

•  Accessible  Raw  Data  and  Results 

•  Robust  and  Comprehensive 

Estimates  can  be  developed  using  a  variety  of  techniques.  There  are 
three  general  approaches. 

1.  Parametric  methods  relate  cost  to  parameters  that  specify  a 
system  within  a  class  of  systems,  such  as,  weight  and  maximum 
speed  for  fighter  aircraft. 

2.  In  estimating  by  analogy  one  adjusts  the  known  costs  of  existing 
systems  similar  to  the  one  in  question  to  arrive  at  cost 
projections. 

3.  Engineering,  or  bottoms-up  estimates  are  made  by  pricing  each 
component  of  a  system,  [p  8-8] 

•  Robust  and  Comprehensive 

•  Simple  In  Concept  and  Approach 

•  Appropriate  Use  of  Analysis 
Resources 

The  cost  input  to  the  analysis  must  be  validated  at  the  sane  level  as 
the  requirements  document  the  analysis  supports.  Validation  should 
identify  the  weaknesses,  or  "soft  areas,"  in  the  cost  estimates,  [p  8-8] 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

•  Accurate  (Accredited  Models  and 
Data;  Trained  and  Experienced 
Analysts  and  Administrators) 

•  Documented 

The  results  of  each  sensitivity  analysis  must  be  documented,  [p  8-9] 

•  Documented 

Cost  estimates  take  into  account  advanced  development  and 
engineering  development,  [p  8-9] 

•  Robust  and  Comprehensive 

•  Accessible  Raw  Data  and  Results 
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TABLE  5.  (Contd.) 


Specified  Expectation  (Page  Numbers  Are  From 

DOD  5000.2M,  Reference  2) 

General  Expectation 

Sensitivity  analyses  show  explicitly  how  military  utility  is  affected 
by  changes  in  system  capability.  They  shown  how  system 
characteristics  (size,  weight,  etc.)  drive  performance,  and  how 
performance  affects  military  utility  or  effectiveness.  Parameters 
should  be  varied  individually  where  it  is  reasonable  to  do  so.  The 
uncertainty  inherent  in  estimating  parameters  and  in  determining 
their  impact  should  be  displayed  explicitly,  [p  8-10] 

•  Accurate  (Accredited  Models  and 
Data;  Trained  and  Experienced 
Analysts  and  Administrators) 

•  Robust  and  Comprehensive 

•  Documented 

Compare  equal-cost  or  equal-effectiveness  alternatives,  [p  8-1 1] 

•  Consistent  (Repeatable, 
Methodological  Integrity) 

Show  the  absolute  values  of  measures.  Make  the  facts  available  and  I 
visible.  Display  the  measures  of  cost  and  effectiveness  for  each 
alternative,  [p  8-11] 

•  Accessible  Raw  Data  and  Results 

•  Documented 

•  Simple  In  Concept  and  Approach 

Managing  The  Administrative  Burden 

One  final  expectation  program  managers  have  in  their  role  as  administrator  of  a 
COEA  is  for  the  COEA  to  be  performed  with  as  light  an  administrative  burden  as 
possible.  Administrative  burden  includes  the  work  involved  with  developing  AirTasks, 
forming  Integrated  Product  Teams,  monitoring  tasks  and  reviewing  products,  as  well  as 
the  wide  range  of  adjudication  and  budgetary  decisions  associated  with  COEAs. 
Adjudication  decisions  include  deciding  whether  to  use  models  or  data  that  are  not 
validated  or  accredited;  deciding  whether  to  fund  a  hardware  upgrade  in  order  to  use  the 
latest  version  of  a  model;  deciding  whether  to  exclude  a  particular  threat  because  the 
existing  models  do  not  include  it  and  deciding  to  negotiate  another  contract  for  analytical 
support  because  the  existing  in-house  supplier  is  over  committed  or  lacks  the  resources  to 
address  particular  assessment  requirements. 
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COMBINING  THE  EXPECTATIONS  OF 
CUSTOMERS  AND  SUPPLIERS 


Since  the  distinction  between  scientist  (the  supplier  of  analysis)  and  administrator 
(the  customer)  is  really  not  definitive  and  their  ultimate  goal  is  the  same,  namely, 
supporting  the  acquisition  of  the  best  system  to  do  the  job,  it  can  be  argued  that  the 
expectations  of  an  enlightened  administrator  derive  from  an  intuitive  appreciation  of  good 
science  with  explicit  recognition  of  the  demands  of  a  schedule  and  a  budget.  So  the 
administrator's  expectations  intrinsically  include  the  scientist’s  expectations  tempered 
with  expectations  imposed  by  schedule  and  budget.  The  following  list  of  expectations  is 
based  on  the  preceding  discussion  of  scientist  and  administrator  needs.  It  is  oriented  to 
the  administrator’s  goal:  to  encourage  and  promote  the  practice  of  good  science  while 
meeting  production  deadlines.  Expect  a  quality  COEA  to  be: 

1.  Consistent  (repeatable,  methodological  integrity) 

2.  Accurate  (accredited  models  and  data;  trained  and  experienced  analysts 
and  administrators) 

3.  Robust,  (comprehensive  results) 

4.  Well  documented 

5.  Affordable 

6.  Timely 

7.  Accessible 

8.  Appropriate  in  the  use  of  analysis  resources 

9.  Simple  in  concept  and  approach  (understandable) 

10.  Produced  with  minimal  administrative  burden. 
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RESOURCES  REQUIRED  TO  MEET  EXPECTATIONS 


As  these  ten  expectations  are  considered  in  detail,  the  practical  requirements  for 
meeting  the  expectation  are  addressed.  In  particular,  the  hardware  (including  the 
associated  resources  of  facilities  and  ranges),  software,  procedures,  and  personnel 
required  to  meet  the  expectation  are  identified. 


CONSISTENCY 


Definition 

Consistency  can  address  two  aspects  of  an  assessment,  namely  the  external 
consistency  of  the  results  of  one  assessment  with  other  related  assessments  and  the 
internal  consistency  of  the  models,  databases,  and  procedures  used  to  produce  a  given 
assessment.  External  consistency  is  usually  addressed  in  terms  of  “repeatability”  while 
internal  consistency  may  be  considered  in  terms  of  “methodological  integrity.” 


Repeatability 

Repeatability — probably  the  most  useful  surrogate  for  external  consistency — is  the 
fruit  of  internal  consistency.  An  assessment  is  said  to  be  repeatable  or  consistent  when 
two  independent  researchers  using  essentially  the  same  tools  and  procedures  and  working 
under  the  same  environmental  conditions  are  able  to  make  the  same  observations;  within 
the  bounds  of  the  statistical  variability  determined  for  the  observations.  Repeatability 
also  addresses  the  situation  where  at  a  later  time  the  same  researcher  conducts  the  same 
assessment  and  makes  the  same  observations. 

Repeatability  is  achieved  through  consistent,  disciplined  use  of  all  the  resources 
involved  in  the  research  and  the  disciplined  accounting  for  all  environmental  factors  that 
may  influence  the  research.  In  addition,  this  disciplined  use  of  resources  involves 
ensuring  that  all  tools  used  in  the  research  are  calibrated  to  certified  standards;  the 
research  is  conducted  under  carefully  controlled  and  monitored  conditions  designed  to 
limit  the  influence  of  intervening  variables,  and  meticulous  records  are  kept  of  the 
calibration  history  of  tools  and  the  results  of  environmental  monitoring,  as  well  as  the 
values  assigned  to  the  variables  under  assessment,  and  the  observations  made. 


Methodological  Integrity 

Meeting  the  expectation  for  internal  consistency  involves  meeting  the  customer's 
need  to  be  assured  the  criteria  used  to  select  tools,  databases,  hardware  and  procedures 
were  applied  uniformly  throughout  the  entire  analysis  process.  This  internal  consistency 
is  the  key  to  using  assessments  in  conjunction  with  other  analysis  and  in  supporting  the 
repeatability  and  external  consistency  of  the  assessment. 
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Consistency  Means  Credibility 

The  significance  of  consistency  in  the  research  arena  is  that  it  is  the  primary  criterion 
used  to  determine  the  credibility  of  any  conclusions  drawn  based  on  the  research.  Thus, 
the  credibility  of  the  research  depends  on  the  consistent  application  of  the  discipline  of 
the  scientific  method  throughout  the  research,  and  it  is  this  same  constancy  of  application 
that  customers  expect  of  assessments  performed  in  support  of  the  acquisition  process. 
Customers  must  be  assured  analysts  are  consistent  in  the  approach  they  take  throughout 
the  assessment;  in  the  way  they  select  databases;  in  the  way  they  select  and  use  models; 
in  the  way  they  employ  supporting  analysts;  in  the  way  they  report  their  results;  and  in 
the  level  of  internal  review  to  which  assessments  are  subjected. 

There  are  numerous  techniques  for  meeting  and  exceeding  the  expectation  for 
consistency.  These  techniques  involve  the  way  hardware,  software,  procedures,  and 
people  are  employed  in  the  assessment  process. 


Hardware  To  Achieve  Consistency 

The  most  obvious  way  to  achieve  consistency  in  the  use  of  hardware  is  to  limit  or 
carefully  control  the  types  and  quantities  of  hardware  used.  Since  assessments  often 
involve  the  use  of  computers  (and  associated  software)  to  model  weapon  systems,  a  very 
direct  way  to  achieve  consistency  in  the  assessment  process  is  to  use  a  single  computer 
and  operating  system.  If  such  a  direct  approach  is  not  possible,  then  every  effort  should 
be  made  to  minimize  the  number  of  computer  types  and  operating  systems  used  in  the 
analysis.  In  any  event,  the  analyst  should  establish  a  set  of  criteria  by  which  computer 
types  and  operating  systems  are  selected  and  those  criteria  should  be  consistently  applied 
throughout  the  analysis  process. 


Software  To  Achieve  Consistency 

As  with  computer  hardware,  the  most  direct  approach  of  using  software  to  achieve 
consistency  in  assessments  is  to  limit  or  carefully  control  the  software  used.  In  the 
simplest  case,  this  involves  using  the  same  model  and  the  same  database  throughout  the 
assessment.  Currently,  the  state-of-the-art  in  modeling  software  does  not  support  a  single 
model  that  can  address  all  aspects  of  a  mission-effectiveness  analysis  for  a  complex 
weapon  system.  However,  with  the  improvements  projected  for  both  hardware  and 
software,  a  single  model  architecture  that  allows  for  the  integration  of  multiple  models 
may  soon  be  available. 


Procedures,  Processes,  And  Methodologies  To  Achieve  Consistency 

Along  with  the  judicious  use  of  hardware  and  software,  the  use  of  various  procedural 
devices  is  likely  to  promote  consistency  and  the  associated  repeatability  in  assessments. 
The  procedure  of  applying  configuration  management  to  control  the  software  and 
hardware  involved  in  analysis  is  critical  to  reproducing  the  particular  configuration  of 
hardware  and  software  employed  in  making  an  assessment,  and  thus,  a  key  to 
reproducing  the  assessment. 

One  of  the  most  effective  and  commonly  used  procedures  for  achieving  consistency 
and  supporting  reproducibility  is  the  careful  documentation  of  all  aspects  of  the 
assessment.  NAWCWPNS  has  developed  a  hierarchy  of  documentation  guides 
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(References  11  and  12)  that  can  be  used  to  report  on  all  aspects  of  assessments. 
Documents  prepared  in  accordance  with  these  guides  must  pass  strict  standards  of 
formatting  and  review  and  should  provide  all  the  information  needed  to  reproduce  a  test 
or  assessment. 


Personnel  Issues  Affecting  Consistency 

Finally,  meeting  the  expectation  for  consistency  involves  a  thorough  understanding 
on  the  part  of  both  administrators  and  analysts  of  the  key  role  played  by  people  in 
achieving  consistent  results.  Both  administrators  and  analysts  should  be  trained  in  the 
fundamentals  of  the  scientific  method  and  the  design  of  experiments.  Analysts  should  be 
trained  in  the  use  of  particular  models  and  databases  they  use  and  the  hardware  associated 
with  their  use.  They  should  be  familiar  with  the  local  analysis  process,  as  well  as  the 
fundamentals  of  the  weapon  system  acquisition  process.  Administrators  should  be 
thoroughly  trained  in  the  weapon  systems  acquisition  process  and  the  fundamentals  of  the 
analysis  process.  Administrators  and  analysts  leading  the  assessment  effort  should  be 
experienced. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  consistency  is  presented  in  Table  6. 

TABLE  6.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Consistency  in  Analysis. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  consistency 


Hardware 


Software 


Procedure,  process, 
methodology 


Personnel 


Same  model 


Limit  types  and 
quantities  of 
hardware  used 

Minimize  types  of 
computers  and 
operating  systems 


Same  model  version 

Same  data 

Minimize  models  and 
databases 

Configuration 

management 

software 

Model  integrating 
architecture 


Configuration  manage¬ 
ment  in  accordance 
with  military 
standards 

Department/Center 
review  of  assessment 
tasks  and  products 

Apply  documentation 
standards  to  models, 
databases,  and 
analysis  products 

Assessment  process 


Analysts  trained  in  use 
of  models,  databases, 
and  computers;  local 
analysis  process; 
fundamentals  of 
weapon  system 
acquisition 

Administrators  trained 
in  weapon  system 
acquisition  process; 
fundamentals  of 
analysis 

Training  in  the  scientific 
method  and  design  of 
experiments 


ACCURACY 


Definition 

The  most  accurate  assessment  of  a  weapon  system  involves  measuring  the 
effectiveness  of  the  actual  system  in  the  combat  environments  for  which  it  is  designed. 
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Under  such  circumstances  the  system  is  used  by  field  personnel  against  an  active  threat. 
Duplicating  these  conditions  is  the  goal  of  the  Operational  Test  and  Evaluation 
(OPEVAL)  required  of  all  new  acquisitions.  Along  with  OPEVAL  assessments  is  the 
Live  Fire  Test  and  Evaluation  (LFT&E)  process  designed  to  evaluate  actual  hardware  in 
the  presence  of  a  carefully  controlled  threat. 

As  a  weapon  system  moves  through  the  phases  of  the  acquisition  process,  the  degree 
to  which  it  can  be  accurately  assessed  changes.  Initially  it  is  impossible  to  evaluate  the 
hardware  against  actual  threats  because  there  is  no  hardware  and  the  threat  may  be  no 
more  than  an  intelligence  projection.  Under  these  conditions,  simulations  and  models  are 
the  appropriate  tools  for  assessing  the  proposed  weapon  system. 


Hardware  To  Achieve  Accuracy 

Running  detailed  engineering  models,  hardware-in-the-loop  models,  operator-in-the- 
loop  models,  and  handling  large  databases  and  model  integration  requires  high-capacity, 
high-speed  computers.  It  requires  computers  that  can  be  linked  with  hardware  in  the 
laboratory  or  in  the  field,  operators  in  the  laboratory  or  in  the  field,  and  facilities  where 
multidimensional  analysis  may  be  performed  (e.g.,  digital  computer  modeling  along  with 
hardware-in-the-loop  simulations). 


Procedures,  Processes,  And  Methodologies  To  Achieve  Accuracy 

To  achieve  accuracy  when  using  models  to  perform  analysis,  the  models  and 
associated  databases  must  accurately  represent  the  objects  under  evaluation,  e.g.,  the 
threat  missile  system  and  the  aircraft  system  modeled  in  a  survivability  assessment,  as 
well  as  the  environment  in  which  the  objects  are  expected  to  operate.  Models  and 
databases  are  said  to  meet  these  conditions  when  they  have  been  “verified”  and 
“validated.”  Verification  involves  testing  models  to  determine  if  the  mathematical 
algorithms  used  to  represent  the  weapon  system,  the  environmental  system,  and  the  threat 
system  really  do  model  these  systems  over  the  range  of  input  data  considered  appropriate 
for  the  model.  Verifying  a  model  is  accomplished  by  running  a  test  case  for  which  the 
outputs  have  been  calculated  and  confirmed  using  previously  verified  models,  proven 
subroutines,  or  “hand”  calculations.  The  results  of  this  run  are  compared  to  the 
confirmed  results  and  the  model  is  verified  to  be  correct  if  it  produces  the  same  result  as 
the  proven  techniques.  Since  models  are  often  complex — involving  thousands  of  lines  of 
computer  code  and  hundreds  of  mathematical  algorithms — the  key  to  verification  is 
developing  and  using  test  cases  that  exercise  the  algorithms  over  the  entire  range  for 
which  they  are  designed.  Furthermore,  test  cases  must  be  subjected  to  the  same 
configuration  control  as  any  other  software  to  ensure  new  model  versions  can  be 
compared  to  their  predecessors. 

Validating  a  model  involves  comparing  the  output  of  the  model  to  empirical  data. 
The  empirical  data  to  which  the  model  results  are  compared  may  be  developed  in 
laboratory  tests,  field  tests,  the  exploitation  of  hardware,  or  possibly  from  actual  combat. 
In  any  event,  for  these  data  to  be  comparable  to  the  model  output,  the  object  undergoing 
the  field  test  must  resemble  the  object  being  modeled.  This  means  validation  is  usually 
limited  for  models  of  new  systems  and  models  are  seldom  validated  for  the  full-range  of 
conditions  under  which  the  weapon  system  might  be  used.  Regardless  of  these 
operational  limits  to  validation,  the  customer  expects  assessment  results  to  reflect  actual- 
test  and  field-use  results  as  accurately  as  possible. 
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Additional  procedures  include  the  review  of  model  results  by  technical  experts 
associated  with  the  kind  of  modeling  performed,  and  procedures  for  analysts  to  perform 
quality  checks  on  the  data  they  are  using  and  producing. 


Software  to  Achieve  Accuracy 

Given  a  model  that  can  be  verified  and  validated,  achieving  accuracy  in  modeling  a 
complex  weapon  system  in  the  full-spectrum  environment  in  which  it  may  be  used 
implies  the  need  for  software  that  links  models  to  hardware;  provides  for  operator-in-the- 
loop  assessments;  provides  for  detailed  modeling,  efficient  modeling,  and  modeling  that 
minimizes  the  need  for  operator  monitoring  or  the  production  and  handling  of 
intermediate  results.  The  concept  of  a  model  integrating  architecture,  mentioned  earlier, 
is  a  response  to  these  software  needs.  It  addresses  the  need  for  a  full  range  of  modeling 
from  detailed  “emulative”  models  to  the  less  detailed  many-on-many  mission- 
effectiveness  models.  Such  an  architecture  also  addresses  the  need  for  efficiency  in 
handling  data  within  and  between  models,  hardware,  operators,  and  databases. 


Personnel  Issues  Affecting  Accuracy 

Analysts  involved  in  producing  accurate  assessments  should  be  trained  and 
experienced  in  the  operation  of  the  models  and  data  they  use.  Where  they  are  responsible 
for  the  detailed  assessment  of  weapon  system  components,  the  analyst  should  be  familiar 
with  the  weapon  system  under  evaluation  and  with  the  design  and  operation  of  the 
component.  Administrators  and  analysts  should  be  trained  in  the  performance  of 
integrated  assessments,  including  the  tasks  involved  with  coordinating  hardware,  field 
tests,  facilities,  operators,  and  models. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  accuracy  is  presented  in  Table  7. 

TABLE  7.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Accuracy  in  Analysis. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  accuracy 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

High  capacity 

High  speed 
Hardware-in-the-loop 
Operator-in-the-loop 
Special  facilities 

Networked  hardware, 
operators,  facilities 

Exploited  hardware 

Emulation  quality 
software 

Minimize  data  handling 

Software  to  link 
hardware,  facilities, 
and  operators 

Model  integrating 
architecture 

W&A 

Standard  test  cases 

Analysts  and  supervisors 
perform  quality 
checks 

Analysis  reviewed  by 
technical  experts 

Analysts  trained  in 
specific  types  and 
components  of 
weapon  systems 

Training  in  performing 
integrated,  complex 
assessments 

Training  in  the  use  of  all 
analytical  tools 
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ROBUST  ASSESSMENTS 


Definition 

Robust  assessments  are  developed  from  analysis  that  is  insensitive  to  reasonable 
variations  in  the  characteristics  of  the  weapon  system,  threat,  environment,  and  any  other 
conditions  pertinent  to  the  evaluation.  Assessments  are,  and  should  be,  subjected  to 
penetrating  criticism  that  inevitably  includes  questions  regarding  the  effectiveness  of  the 
candidate  system  with  a  range  of  characteristics  against  a  range  of  threats  in  a  range  of 
scenarios. 

Providing  assessments  that  address  these  questions  involves  a  careful  and  thorough 
definition  of  the  weapon  system,  the  threat,  and  the  environment.  As  the  weapon  system 
under  evaluation  is  modeled  in  terms  of  critical  performance  characteristics,  the  analyst 
must  not  only  justify  the  values  assigned  to  these  characteristics,  but  also  the  range  over 
which  the  values  are  varied.  Likewise,  the  analyst  must  evaluate  the  system  against  the 
widest  possible  range  of  threats  in  the  widest  possible  range  of  environments  using  the 
widest  possible  range  of  tactics.  Furthermore,  a  robust  assessment  will  address 
performance  in  a  comprehensive  mission  environment  where  the  system  is  employed  as 
part  of  a  tactical  unit  interacting  with  other  tactical  units,  the  threat,  and  the  natural 
environment. 

Robust  assessments  are  also  the  product  of  analytical  diversity.  An  assessment  is 
robust  when  the  independent  results  of  numerous  types  of  evaluations  lead  to  the  same 
general  conclusions.  For  example,  this  would  be  the  case  when  the  results  of  an  analysis 
performed  with  a  digital  computer  model  were  reinforced  with  the  results  of  independent 
field  or  laboratory  tests  of  the  same  or  similar  system. 


Hardware  To  Achieve  Robust  Assessments 

Hardware  and  facilities  supporting  robust  assessments  include  the  full  spectrum  of 
laboratory  test  equipment,  digital  computers,  test  ranges,  hardware-in-the-loop  and 
operator-in-the-loop  equipment  and  facilities,  communication  and  data  links  between 
hardware  and  facilities,  exploited  hardware,  and  the  means  for  exploiting  tests  and 
hardware. 


Software  To  Achieve  Robust  Assessments 

Software  supporting  robust  assessments  includes  models  that  accurately  represent  the 
critical  characteristics  of  the  system  under  evaluation,  the  threat,  and  the  environment 
over  as  wide  a  range  as  possible.  The  models  should  also  provide  a  comprehensive 
representation  of  the  mission  context  in  which  the  system  is  to  be  employed.  This 
includes  a  complete  representation  of  the  natural  and  man-made  environments  and  the 
command,  control,  communications,  and  intelligence  (C^I)  network.  Robust  assessments 
also  call  for  software  that  provides  for  interactive  modeling  using  operators-in-the-loop 
and  hardware-in-the-loop  and  the  linking  of  models  to  field  and  laboratory  tests. 
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Procedures,  Processes,  And  Methodologies  To  Achieve 
Robust  Assessments 

Examples  of  the  procedural  techniques  that  may  be  employed  to  supply  robust 
assessments  are  the  development  and  use  of  plausible  and  comprehensive  scenarios; 
interim  reviews  between  analysis  suppliers  and  customers;  use  of  verified  and  validated 
models;  coordination  of  field  tests,  laboratory  tests  and  analysis  runs;  and  the 
consolidation  of  current  analysis  and  testing  with  related  analysis  previously  performed. 


Personnel  Issues  Affecting  Robust  Assessments 

To  use  the  hardware,  software,  and  procedural  techniques  identified,  analysts  and 
administrators  need  to  be  trained  in  military  operations  in  order  to  appreciate  the  variety 
of  conditions  in  which  weapon  systems  must  operate.  Participation  in  field  exercises  and 
activities  such  as  environmental,  reliability,  and  maintainability  testing  may  also  provide 
valuable  insight  into  the  mission  environment.  Analysts  should  also  be  cross-trained  in 
the  use  of  different  types  and  levels  of  models  and  field  and  laboratory  testing. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  robust  assessments  is  presented  in  Table  8. 

TABLE  8.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Robust  Assessments. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  robustness 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Hardware-in-the-loop 
Operator-in-the-loop 
Exploited  hardware 
Range  tests 

Field/fleet  exercises 

Networked  hardware 

"Mission"  analysis 
capability 

Model  full  spectrum  of 
natural  and  man-made 
environment 

Model  C3I  links 

Interactive 

W&A 

Standard  scenarios 

Customer/supplier 

reviews 

Assessment  process 

Administrators  and 
analysts  trained  in 
military  operations 

Participation  in  field 
exercises 

Analysts  cross-trained  in 
weapon  system  types 
and  tactics 

WELL-DOCUMENTED  RESULTS 


Documentation  Issues 

An  assessment  is  not  complete  until  it  has  been  documented  and  published  in  a 
timely  manner.  In  effect,  the  assessment  report  is  the  tangible  result  of  the  assessment 
and  it  should  be  recognized  and  treated  as  such.  Customers  should  carefully  specify  the 
kind  of  report  or  reports  they  expect  and  analysts,  as  good  scientists,  should  be  eager  to 
supply  a  report  that  presents  their  findings  clearly,  provides  the  detail  necessary  to 
reproduce  and  confirm  their  results,  and  supports  further  assessments.  A  good  summary 
of  the  issues  that  pertain  to  the  expectations  associated  with  documentation  is  provided  in 
the  Independent  Cost  Analysis  section  of  DOD  Manual  5000.2M  (Reference  3,  p  15-3). 
According  to  the  manual, 
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“The  purpose  of  the  documentation  of  cost  estimates  is  to  provide  sufficient  information  about 
the  way  the  estimates  were  produced  so  that  Cost  Analysis  Improvement  Group  analysts 
could,  provided  access  to  the  databases  employed,  reproduce  the  estimates.  The  means  by 
which  each  part  of  the  estimate  was  produced  must  be  fully  explained. 

Where  a  cost  estimating  relationship  is  used,  its  source  must  be  cited  completely,  or  the  model 
and  the  set  of  data  with  which  it  was  calibrated  must  be  cited. 

Where  judgment  was  used  to  adjust  estimates  made  by  analogy  with  other  systems  or 
components  of  systems,  the  backgrounds  of  those  making  the  judgment  must  be  given  (e.g., 
are  they  cost  analysts,  engineers,  vendor  or  Government  employees?)  as  well  as  complete 
citations  of  the  sources  of  the  costs  of  the  analogous  system(s).  Sources  of  the  costs  of  each 
element  in  an  engineering  or  ‘grass  roots’  estimate  must  be  cited  completely.” 

Also  of  critical  importance  regarding  the  issue  of  documentation  is  the  role  it  plays 
as  . .  a  historical  record  of  the  alternatives  considered  at  each  milestone  decision  point” 
(Reference  2,  p  4-E-l). 


Hardware  Supporting  Documentation 

The  documenting  of  an  assessment  can  be  facilitated  by  the  availability  of  hardware 
such  as  personal  computers,  portable  (lap-top)  computers,  high-resolution  printers  and 
plotters,  and  other  computer  peripherals  for  displaying  results  such  as  scanners  and  video- 
to-digital  converters.  Other  than  hardware  dedicated  to  documentation,  consideration 
should  be  given  to  computer  workstations  or  main  frames  that  can  host  quality  word¬ 
processing  software. 


Software  Supporting  Documentation 

High-quality  documentation  often  requires  a  combination  of  word  processing,  spread 
sheet,  presentation,  computer-aided  design  (CAD),  and  database  software.  In  addition, 
models  with  such  built-in  software  such  as  statistical,  documentation,  and  graphics  post¬ 
processors  support  the  documentation  process.  Since  report  formatting  is  a  very 
demanding  yet  routine  part  of  documentation,  some  kind  of  software  package  addressing 
local  report-formatting  requirements  would  likely  enhance  the  documentation  process. 


Procedures,  Processes,  And  Methodologies  Supporting  Documentation 

Since  documentation  is  the  tangible  result  of  an  assessment,  administrators  at  the 
local  level  should  include  it  as  an  expected  result  in  the  analyst’s  performance  plan. 
Administrators  should  be  prepared  to  review  or  arrange  for  the  review  of  documentation 
produced  both  in  terms  of  clarity  and  technical  accuracy.  Analysts  should  be  able  to 
reference  databases  and  models  in  their  reports;  this  presents  a  requirement  that  databases 
and  models  be  documented.  A  standard  documentation  format  for  assessments  facilitates 
the  documentation  and  review  process,  and  a  standard  review  process  in  turn  establishes 
guidelines  for  documentation. 
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Personnel  Issues  Affecting  Documentation 

Well  designed  and  executed  assessments  address  documentation  as  an  integral  part 
of  the  assessment.  In  terms  of  training,  this  suggests  the  need  for  analysts  and  mid-level 
administrators  to  be  competent  in  the  design  of  experiments,  and  in  the  skills  involved  in 
data  reduction  and  data  presentation  such  as  statistics  and  graphics.  Analysts  should  be 
familiar  with  document  formatting  requirements  and  review  processes.  Administrators 
supervising  analysts  should  be  trained  in  performance  planning  and  in  reviewing 
documentation 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  well  documented  assessments  is  presented  in  Table  9. 


TABLE  9.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Well  Documented  Analysis. 


Type  of  resources  required  to  meet  and  exceed  expectation  of  well  documented  analysis 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Personal 

Word  processing, 

Document  review 

Analysts  trained  in 

computers 

spread  sheet,  and 

Performance  plans  requiring 

statistics, 

Workstations 

presentations 

documentation  of  analysis 

documentation,  design 

with  capacity 

software 

to  meet  expectations 

of  experiments,  and 

for  word 

Word-processing 

Standard  documentation 

local  and  headquarters 

processing  and 

software  tailored  to 

format 

analysis  review 

graphics 

TP  or  TM  format 
Graphics  outputs 
Statistics  post¬ 
processors 

Models  with 
documentation  post¬ 
processors 

Level  of  documentation 
linked  to  level  of  analysis 
Documented  databases 

Documented  software 
Assessment  process  that 
includes  documentation 

processes 

Administrators  trained  in 
documentation  review 
and  performance 
planning 

AFFORDABILITY 


Affordability  Issues 

Cost  constraints  limit  the  amount  of  analysis  and  testing  included  in  an  assessment, 
but  they  need  not  limit  meeting  and  exceeding  customer  expectations  for  a  quality 
assessment.  Too  often  cost  limitations  are  used  as  an  excuse  to  completely  ignore  or  at 
least  dilute  other  expectations  such  as  robustness  or  documentation.  The  object  is  to 
make  quality  assessments  affordable  by  recognizing  at  the  outset  of  the  process  that 
resources  are  limited  and  the  assessment  should  be  made  in  accordance  with  those 
limitations. 
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Hardware  Promoting  Affordability 

Hardware  contributes  to  the  cost  of  assessments  in  two  ways:  first  as  an  initial  capital 
outlay,  then  as  an  item  requiring  maintenance  and  other  overhead  support.  In  addition  to 
these  costs,  acquiring  and  using  hardware  involves  paying  and  training  operators  and 
these  costs  should  be  considered  in  the  acquisition  and  use  of  the  hardware.  Means  for 
reducing  these  costs  include  limiting  the  amount  of  hardware  needed  for  an  assessment, 
sharing  hardware  with  other  assessment  projects,  acquiring  hardware  with  low  overhead 
expenses,  and  efficiently  using  the  hardware. 


Software  Promoting  Affordability 

Software  used  in  performing  an  assessment  supports  affordability  when  it  enables 
analysts  to  perform  their  jobs  quickly,  allows  for  the  efficient  integration  of  supporting 
analysis  such  as  field  tests  and  hardware-  or  operator-in-the-loop  assessments,  and 
supports  the  documentation  and  configuration  management  tasks.  Such  software  has  the 
characteristics  of  ease  of  use  (user  friendly),  efficient  architecture,  flexibility  (i.e., 
software  that  can  be  used  on  different  types  of  hardware  and  software  that  supports  the 
assessment  of  more  than  a  single  aspect  of  an  engagement),  and  low  overhead  in  terms  of 
licensing  fees,  upgrading,  and  training.  Software  that  is  verified,  validated,  and  well 
documented  supports  affordability  by  relieving  the  user  of  these  responsibilities  in  the 
course  of  performing  the  assessment.  Of  considerable  aid  to  administrators  when 
accounting  for  the  cost  of  performing  the  assessment,  would  be  a  networked  program 
management  software  package  enabling  them  to  track  all  analytical  resources  consumed. 
Likewise,  analysts  could  use  some  form  of  accounting  software  to  give  them  an  up-to- 
the-minute  status  of  all  the  cost-related  parameters  associated  with  their  models.  Such 
software  would  allow  for  the  development  of  metrics  to  improve  their  models  and 
analytical  processes.  Since  many  of  these  software  characteristics  match  those  set  as 
goals  for  a  model  integrating  architecture,  it  seems  that  such  an  architecture  also  promises 
a  means  for  more  affordable  assessments. 


Procedures,  Processes,  And  Methodologies  Promoting  Affordability 

Procedural  steps  for  achieving  affordability  include  optimizing  hardware  usage  with 
such  procedures  as  those  that  ensure  the  number  of  computer  runs  made  for  an  analytical 
task  is  appropriate  for  the  type  of  model  being  used,  that  ensure  the  fidelity  of  the  model 
and  data,  and  that  clearly  define  the  requirements  for  statistical  reduction  of  the  data. 
Procedures  may  also  aid  in  optimizing  and  coordinating  the  number  of  suppliers  involved 
in  the  assessment  and  in  optimizing  and  coordinating  the  use  of  expensive  resources  such 
as  sophisticated  operator  or  hardware-in-the-loop  assets,  range  time,  laboratories, 
specialists,  and  field  exercises. 


Personnel  Issues  Affecting  Affordability 

Administrators  and  analysts  need  to  be  aware  of  how  the  various  elements  of 
assessments  contribute  to  the  cost  of  the  assessment.  This  suggests  a  need  for 
administrators  and  analysts  to  be  familiar  with  the  types  of  assets  used  to  perform 
analysis  and  the  costs  associated  with  acquiring,  using,  and  maintaining  those  assets. 
Both  administrators  and  analysts  need  to  be  well  trained  and  experienced  in  the  practice 
of  their  respective  disciplines. 
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A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  affordable  assessments  is  presented  in  Table  10. 


TABLE  10.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Affordable  Analysis. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  affordable  analysis 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Low  overhead 

Low  overhead 

Minimize  computer 

Administrators  trained 

Fast 

Fast 

usage 

in  the  cost  of  owning 

Networked  hardware 

Flexible 

Optimize  hardware-in- 
the-loop  and/or 

and  operating 
simulation  and  test 

Reliable 

Program  management 

interactive 

capabilities 

Short  repair  and/or 

software 

requirements 

Analysts  trained  in  the 

maintenance  time 

On-line,  up-to-date 

Minimize  number  of 

proficient  use  and 

Supports  multiple  users 

management 
information  system 

Budget  presentation  and 
analysis  software 

suppliers 

"Tag  along" 
arrangements  with 
field  tests 

maintenance  of  their 
models 

Administrators  and 
analysts  certified  in 
their  disciplines 

TIMELINESS 


Timeliness  Issues 

When  the  effectiveness  of  a  military  force  is  linked  more  to  the  quality  rather  than 
the  quantity  of  the  weapon  systems  employed,  the  acquisition  of  new  systems  is  driven  by 
a  need  to  employ  new,  effectiveness-enhancing  technologies  as  soon  as  possible.  The 
more  a  force  depends  on  the  qualitative  superiority  of  its  weapons  to  achieve  superiority 
on  the  battlefield,  the  more  critical  is  the  time  factor  in  weapons  acquisition. 

Since  the  effectiveness  of  the  U.S.  military  does  depend,  in  part,  on  weapon  systems 
that  are  technically  superior  to  those  of  potential  adversaries,  there  is  a  constant  need  to 
infuse  new  technologies  into  the  weapons  systems  employed  by  our  forces.  This  demand 
for  fresh  technologies  is  a  primary  reason  acquisition  managers  expect  timeliness  in  the 
performance  of  assessments  supporting  the  weapon-system  acquisition  process.  Of 
course,  there  are  other  reasons  to  perform  assessments  quickly.  These  include  the  need  to 
keep  the  cost  of  the  assessment  down  and  the  need  to  perform  assessments  within  the 
scheduling  constraints  associated  with  limited  assets  such  as  ranges,  specialized  facilities 
and  laboratories,  computers,  and  personnel.  Furthermore,  the  performance  of 
assessments  must  be  coordinated  with  other  customers  and  suppliers,  maintenance  cycles, 
training,  and  in  the  case  of  an  asset  such  as  a  test  range,  even  the  weather.  For  all  these 
reasons,  time  is  a  critical  element  of  assessments. 
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Hardware  Promoting  Timeliness 

Hardware  designed  to  support  timely  assessments  can  be  quickly  set  up  for  its 
assessment  function,  is  easy  to  use,  has  minimal  maintenance  or  other  downtime 
requirements  (high  reliability,  low  failure  rate,  short  repair  time),  operates  in  all-weather 
and  laboratory  conditions  (for  hardware  exposed  to  such  conditions),  supports  multiple 
users  simultaneously,  supports  time-saving  software,  supports  the  automatic  flow  of  data 
from  other  hardware,  and  performs  its  function  quickly. 


Software  Promoting  Timeliness 

Software  features  supporting  timely  assessments  include  high  speed  (optimum 
architecture  and  languages);  user  friendliness;  and  automated  links  between  databases, 
other  models,  hardware,  documentation,  and  configuration  management  software.  To 
support  timely  assessments,  software  should  be  verified,  validated,  and  documented.  To 
as  great  an  extent  as  possible,  software  should  be  quickly  and  easily  transportable 
(hardware  independent)  so  analysts  using  various  types  of  hardware  can  be  using  the 
same  model  simultaneously  and  data  generated  or  stored  on  one  type  of  hardware  can  be 
used  with  models  loaded  on  other  types  of  hardware.  Again,  many  of  these  software 
features  match  those  set  as  goals  for  a  model  integrating  software  architecture,  so  it  seems 
such  an  architecture  also  has  merit  as  a  means  to  more  timely  assessments.  To  aid 
administrators  in  accounting  for  the  time  spent  in  performing  the  assessment,  a  networked 
program  management  software  package  would  be  helpful.  In  the  same  vein,  analysts 
could  use  software  to  give  them  an  up-to-the-minute  accounting  of  all  the  time  related 
parameters  associated  with  their  models.  Such  software  would  allow  for  the 
identification  and  alleviation  of  "bottle  necks"  in  the  models  and  analytical  processes. 


Procedures,  Processes,  And  Methodologies  Promoting  Timeliness 

Other  means  for  meeting  the  expectation  for  timeliness  in  assessments  include 
procedures  that  help  optimize  the  number  of  suppliers  involved  in  an  assessment  to 
minimize  the  total  time  involved.  This  kind  of  optimizing  needs  to  occur  at  all  levels  of 
the  assessment  process  and  suggests  that  procedures  for  team  building  are  a  valuable  way 
to  make  the  process  run  quicldy.  Other  procedural  means  supporting  timeliness  include 
procedures  to  minimize  the  number  of  runs  associated  with  computer  modeling  and  the 
number  of  trials  associated  with  field  or  laboratory  tests.  Procedures  may  be  useful  in 
minimizing  the  reviews  involved  in  the  documentation  aspect  of  the  assessment,  avoiding 
the  delays  associated  with  training,  verification,  and  validation;  ensuring  analysts  and 
administrators  involved  in  the  assessment  are  properly  experienced  and  trained;  and 
ensuring  software  and  data  are  documented  and  configuration  management  is  practiced. 
Procedures  may  also  be  useful  in  providing  for  the  quality  control  of  analysis  at  the  level 
where  it  is  performed,  while  it  is  being  performed. 


Personnel  Issues  Affecting  Timeliness 

In  order  to  provide  timely  assessments,  administrators  should  be  experienced  and 
well  trained  in  the  weapon  system  acquisition  process  and  this  includes  skill  in  the  use  of 
program  management  software.  Administrators  and  analysts  should  appreciate  why  time 
is  such  a  critical  factor  in  the  acquisition  of  high-tech  weapon  systems  for  a  military  force 
dependent  on  qualitative  superiority.  They  should  be  familiar  with  the  hardware, 
software,  and  procedural  means  for  reducing  the  time  involved  in  making  assessments. 
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Analysts  should  be  experienced  and  well  trained  in  the  use  of  the  particular  tools 
associated  with  their  parts  of  the  assessment  and  they  should  be  aware  of  the  need  to 
coordinate  and  expedite  the  flow  of  data  throughout  the  assessment  process.  Analysts 
and  administrators  should  be  prepared  to  quickly  form  and  operate  as  teams. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  timely  assessments  is  presented  in  Table  11. 


TABLE  11.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Timely  Analysis. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  timely  analysis 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Fast  (parallel,  optical) 
User  friendly 

Flexible 

Networked  hardware 

Short  repair  and/or 
maintenance  time 

Supports  multiple  users 

Fast 

Optimum  architecture 

Model  integrating 
architecture 

Program  management 
software 

On-line,  up-to-date 
management 
information  system 

Minimize  number  of 
suppliers 

Minimize  model  runs 

Minimize  reviews 

Provide  quality  control 
at  analysis  level 

Coordinated  customer/ 
supplier  interfaces 

Certified  analysts 

Administrators 
proficient  in  weapon 
systems  acquisition 

Analysts  proficient  in 
analysis  process 

ACCESSIBILITY 


Definition 

Part  of  the  weapon  system  acquisition  process  involves  the  development  of  a 
foundation  of  assessments  that  confirm  each  other  and  justify  the  continued  development 
and  ultimate  fielding  of  a  new  weapon  system.  Such  a  foundation  is  necessary  because 
circumstances  surrounding  the  original  operational  requirement  and  statement  of  need  for 
the  weapon  system  may  change  as  the  acquisition  proceeds.  Thus,  the  acquisition 
manager  is  forced  to  not  only  justify  the  passage  from  one  phase  of  development  to 
another,  but  also  to  re-justify  the  need  for  the  proposed  system  given  changed 
circumstances.  One  means  of  supporting  this  justification  and  re-justification 
requirement  has  already  been  addressed,  namely,  the  production  of  robust  assessments; 
assessments  that  remain  valid  over  a  wide  range  of  developmental  and  operational 
conditions.  Regardless  of  whether  or  not  an  assessment  is  robust,  any  assessment  is  of 
little  use  to  the  acquisition  manager  involved  in  the  justification  or  re-justification  of  the 
system  if  that  assessment  is  not  accessible.  Thus,  accessibility  is  critical  to  the  support  of 
the  weapons  acquisition  process. 

In  addition  to  this  demand  for  accessibility  due  to  the  nature  of  the  acquisition 
process  as  an  accumulation  of  analysis,  there  is  the  demand  for  accessibility  stemming 
from  the  nature  of  assessments  themselves  as  iterative  or  open-ended  products.  As  a 
weapon  system  is  refined  and  modified  in  the  acquisition  process,  assessments  can  be 
refined  to  incorporate  new  levels  of  detail  and  the  results  of  empirical  evaluations.  Model 
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verification,  validation,  and  enhancement  is  occurring  in  parallel  with  the  design  of 
weapon  system  hardware,  so  even  if  the  hardware  changes  little,  the  ability  to  model  it 
may  improve  dramatically  and  warrant  a  reassessment  of  the  system.  Again,  robust 
assessments  made  with  accurate  models  of  well-defined  systems  help  keep  the  need  for 
reassessments  at  a  minimum,  but  for  any  reassessment  to  be  effective  the  original 
assessment  must  be  accessible. 


Hardware  Supporting  Accessibility 

With  the  flexible  hardware  available  today,  accessibility  should  not  be  a  hardware 
problem.  However,  hardware  can  play  an  important  role  in  facilitating  access.  Some 
ways  to  facilitate  access  are  the  acquisition  and  use  of  computers  with  a  capacity  for  the 
storage  and  retrieval  of  the  voluminous  data  generated  in  assessments.  The  hardware 
should  also  support  networking,  so  stored  data  is  accessible  to  multiple  assessment 
suppliers  and  customers.  Given  the  sensitivity  of  some  data  generated,  the  hardware 
should  also  support  the  control  and  monitoring  of  access  to  the  data. 


Software  Supporting  Accessibility 

Software  supports  accessibility  when  it  provides  for  the  automated  storage  and 
cataloging  of  data  as  it  is  generated,  when  it  provides  for  the  automation  and 
standardization  of  data  labeling,  when  the  models  generating  data  have  input  and  output 
interfaces  that  provide  for  direct  linkage  to  data  storage,  when  stored  data  is  formatted  for 
direct  use  by  assessment  models,  and  when  the  software  supports  complete  and  timely 
documentation  of  assessment  results.  Software  that  enhances  accessibility  should  also  be 
user  friendly. 


Procedures,  Processes,  And  Methodologies  Supporting  Accessibility 

The  most  direct  way  to  provide  accessibility  to  assessments  is  to  ensure  the 
assessment  is  completely  documented,  published,  and  distributed  as  soon  as  possible  after 
the  completion  of  the  assessment.  Although  the  generation  of  hard-copy  documents  has 
been  made  somewhat  obsolete  by  the  availability  of  magnetic  and  optical  storage  and 
dissemination,  the  principle  of  documenting  as  the  key  to  accessibility  remains. 
Procedures  for  documentation  should  ensure  that  preparation  and  review  of 
documentation  is  conducted  with  accessibility  requirements  in  mind.  Accessibility 
should  also  be  a  critical  design  criterion  in  the  development  of  models  and  databases. 

Other  procedures  supporting  accessibility  are  the  use  of  documented  software, 
configuration  control  of  software,  making  resources  available  to  develop  and  maintain 
libraries  where  data  are  stored  and  from  which  data  are  retrieved,  providing  for 
networking  among  assessment  archives  and  data  libraries,  providing  for  the  control  and 
monitoring  of  access  to  the  libraries,  and  the  establishment  of  national  centers  (such  as 
the  Information  Analysis  Centers  (IACs)  for  the  collection  and  dissemination  of  models 
and  databases.  The  “World  Wide  Web”  also  offers  a  vehicle  for  providing  tremendous 
access  to  analysis  provided  appropriate  limits  to  access  can  be  achieved. 
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Personnel  Issues  Affecting  Accessibility 

Administrators  and  analysts  need  to  appreciate  the  need  for  accessibility  and  the  fact 
that  documentation  in  its  various  forms  is  the  primary  means  for  achieving  accessibility. 
They  need  to  be  trained  in  the  development  and  use  of  on-line  libraries  and  the  utility  of 
the  IACs.  Analysts  and  administrators  should  consider  taking  an  active  part  in 
professional  societies  and  user  groups  aimed  at  the  development  and  dissemination  of 
assessment  techniques,  the  results  of  ongoing  assessments,  and  development  of 
specialized  databases  and  models. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  accessibility  is  presented  in  Table  12. 


TABLE  12.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  for  Accessible  Analysis. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  accessible  analysis 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

High-volume,  high- 

User  friendly 

Access  to  data  libraries 

Administrators  trained 

speed  storage  and 

Configuration 

management 

controlled  and 

in  use  of  on-line 

retrieval 

monitored 

libraries 

Secure 

Models  with  post- 

Single  library 

Analysts  trained  in 

Networked  hardware 

processors  for  library 
functions 

Well  defined  I/O 
interfaces 

File  servers 

World  Wide  Web  access 
to  models,  data, 
briefings,  program 
definition 

administration 

Standard  file  naming 
convention 

Resources  dedicated  to 
library  development 
and  maintenance 

library  development 
and  maintenance 

APPROPRIATE  USE  OF  ANALYSIS  RESOURCES 


Issues  Relating  To  The  Use  Of  Appropriate  Analysis  Resources 

The  resources  that  may  be  employed  in  assessments  of  weapon  systems  are  as 
complex  as  the  weapon  systems  themselves.  In  fact,  it  may  be  argued  that  an  assessment 
tool  such  as  an  instrumented  range  is  as  complex  as  the  most  complex  system  (or 
combination  of  systems)  tested  on  that  range.  But  in  choosing  the  resources  (tools, 
facilities,  software)  for  a  particular  assessment,  the  primary  selection  criterion  is 
simplicity — to  select  the  least  complex  resource  or  set  of  resources  that  will  support  an 
assessment  that  unequivocally  answers  all  the  questions  posed  at  the  milestone  review. 
Use  of  this  selection  criterion  is  based  on  the  premise  that  the  most  revealing  assessment 
is  that  which  presents  the  clearest  links  between  the  stimulus  to  the  system  and  the 
response  of  the  system.  Achieving  these  clear  links  requires  the  use  of  tools  that  provide 
an  appropriate  level  and  type  of  stimulation,  measure  the  appropriate  responses,  and 
produce  data  that  is  easy  to  use  and  understand  while  the  operation  of  the  tools  and 
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influence  of  those  tools  on  the  data  is  clearly  understood.  Tools  that  cloud  the 
relationship  between  stimulus  and  response  are  tools  that  (1)  produce  unnecessary 
amounts  or  types  of  data;  (2)  produce  data  requiring  unnecessary  manipulation  such  as 
translating  data  produced  in  specialized  formats  or  languages  into  standard  formats  and 
languages  before  it  can  be  used;  (3)  have  not  been  calibrated,  validated,  or  verified;  (4) 
require  the  use  of  inexplicable  and  subjective  adjustment  factors;  or  (5)  produce  data  that 
is  difficult  to  retrieve,  preserve,  or  interpret. 


Appropriate  Hardware 

The  means  to  meet  the  expectation  that  appropriate  analysis  resources  are  used  in  an 
assessment  include:  providing  a  full  spectrum  of  resources  for  use  in  the  various  levels 
and  types  of  assessments;  providing  hardware  that  meets  such  design  criteria  as  ease  of 
use,  low-maintenance,  interoperability,  and  low-training  requirements;  providing 
hardware  suited  to  software  and  peripherals;  providing  hardware  (including  facilities  and 
ranges)  that  meets  the  environmental  constraints  imposed  on  testing;  and  providing 
hardware  that  can  be  linked  to  other  hardware,  software  or  databases  to  achieve  flexibility 
in  building  up  or  down  in  complexity. 


Appropriate  Software 

Software  as  a  means  of  meeting  expectations  for  the  use  of  appropriate  resources  in 
assessments  include:  using  software  that  has  been  validated  and  verified;  coordinating  the 
fidelity  of  models  and  databases;  using  software  that  is  thoroughly  documented; 
providing  software  that  is  user  friendly;  and  providing  software  that  is  easily  linked  to 
other  software  and  that  can  be  run  on  various  types  of  hardware  designed  to  the 
architectural  standards  that  allow  for  modular  growth  in  complexity. 


Appropriate  Procedures,  Processes,  And  Methodologies. 

It  seems  the  development  and  implementation  of  procedures  has  been  one  of  the  key 
ways  to  meet  expectations  for  the  appropriate  use  of  resources  in  assessments.  Some  of 
these  procedures  are  dissemination  of  model  descriptions  in  catalogues  and  dissemination 
of  information  regarding  test  facilities,  specialized  hardware,  and  major  exercises  through 
catalogues  and  brochures;  development  of  calendars  that  list  upcoming  tests;  dedication 
of  resources  for  the  development,  test,  validation,  verification,  and  maintenance  of 
software;  exchange  tours  for  both  administrators  and  analysts  that  encourage 
understanding  and  communication  between  supplier  and  customer;  and  the  establishment 
of  documentation  requirements  that  address  the  need  for  a  clear,  understandable 
explanation  of  the  use  and  impact  of  the  resources  used  in  the  assessment.  Other  related 
means  include  the  coordination  of  software  development,  maintenance,  validation, 
verification,  and  dissemination  through  national  and  multiservice  organizations  such  as 
the  LACs,  Joint  Technical  Coordinating  Group  for  Munitions  Effectiveness  (JTCG/ME), 
JTCG/AS,  Electromagnetic  Code  Consortium  (EMCC),  and  Joint  Coordinating  Group  for 
Electronic  Warfare  (JCG-EW). 
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Personnel  Issues  Associated  With  The  Appropriate  Use  Of  Analysis  Resources 

To  understand  why  care  should  be  taken  to  use  the  least  complex  resources  when 
making  assessments,  administrators  and  analysts  need  to  be  trained  in  the  scientific 
method  and  the  design  of  experiments.  To  know  what  resources  are  available  to  support 
assessments,  both  administrators  and  analysts  need  to  be  introduced  to  the  full  spectrum 
of  resources,  including  a  description  of  the  fundamental  workings  of  the  resources  and 
examples  of  the  types  of  assessments  for  which  the  resource  is  appropriate.  Finally, 
analysis  suppliers  as  well  as  administrators  need  to  be  aware  of  the  kinds  of  questions 
asked  at  program  review  milestones. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  appropriate  use  of  analysis  resources  is  presented  in  Table  13. 


TABLE  13.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations  Associated  With  The 

Appropriate  Use  Of  Analysis  Resources. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  appropriate  use  of  analysis  resources 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Hardware  suited  to 
software 

Hardware  use 
proportional  to 
analysis  problem 

Hardware,  facilities, 
exercises  sensitive  to 
environmental 
constraints 

Networked  hardware 

State-of-the-art  software 

Coordination  of  model 
and  data  fidelity 

Model  integrating 
architecture 

Program  management 
software 

Catalogue  of  models 

Catalogue  of  facilities 

Calendar  of  exercises, 
tests,  TECHEVALs, 
OPEVALs 

Dedicated  resources  for 
software  development, 
testing,  maintenance, 
validation,  and 
verification 

Software  documentation 

Coordination  of 
software  development 
through  national, 
multiservice  bodies 
(e.g.,  JTCG/AS, 
JTCG/ME,  EMCC, 
LACs,  JCG-EW) 

Administrators  and 
analysts  trained  in 
scientific  method  and 
design  of  experiments 

Administrators  and 
analysts  trained  in  full 
spectrum  of  analysis 
resources 

Analysts  trained  in  the 
weapon  system 
acquisition  process 
and  familiar  with  the 
questions  asked  at 
program  milestone 
reviews 

Headquarters-to-field 
exchange  tours 

Exchange  tours  for 
analysts 

SIMPLICITY:  SIMPLE  IN  CONCEPT  AND  APPROACH 


Issues 

For  the  same  reason  that  tools  used  to  support  assessments  should  enable  both 
customer  and  supplier  to  make  the  clearest  links  between  the  stimulus  to  which  the 
proposed  system  is  subjected  and  the  response  of  the  system,  the  entire  assessment  should 
be  only  as  complex  as  necessary  to  provide  answers  to  the  questions  asked  at  the 
milestone  reviews.  It  is  also  important  to  recognize  that  the  acquisition  process  itself  is 
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based  on  the  emergence  of  an  operational  system  in  stages  beginning  with  theories  and 
projections  and  ending  with  hardware.  This  staging  is  used  to  reduce  the  risks  involved 
in  development  by  controlling  the  introduction  of  complexity.  E.  B.  Wilson  (Reference 
8,  p  36),  in  his  study  of  the  design  of  experiments  notes  that, 

“...it  is  almost  always  desirable  or  even  necessary  to  approach  answers  in  stages,  starting  with 
the  most  idealized  and  simplified  version  possible.  When  this  has  been  unraveled,  more 
general  cases  can  be  attacked.” 

Furthermore,  since  assessments  are  made  to  address  system  behavior  that  is  inevitably 
unpredictable,  the  assessment  approach  should  be  simple  and  flexible  enough  to  allow  for 
the  investigation  of  unpredictable  or  unusual  results  as  the  assessment  proceeds.  But  one 
of  the  most  compelling  arguments  for  simplicity  in  the  design  and  execution  of 
assessments  stems  from  the  need  for  the  assessment  to  be  understood.  As  Morse  and 
Kimball  (Reference  4,  p  10)  point  out, 

“...it  must  be  emphasized  that  the  problem  is  not  completed  when  the  details  of  the  operation 
are  understood  by  the  operations  research  worker  alone.  Since  the  purpose  of  the  analysis  is 
to  provide  the  executive  department  with  a  basis  for  decision,  the  problem  is  successfully 
completed  only  when  the  executive  department  understands  the  essential  parts  of  the 
conclusions  of  the  analysis.” 


Hardware  Promoting  Simplicity 

The  conception  and  design  of  an  assessment  involves  expectations  for  quality 
planning  and  management.  These  expectations  are  not  as  easy  to  meet  with  hardware  as 
some  of  the  other  expectations  addressed.  It  is  likely  that  meeting  the  expectation  for 
simplicity  in  the  conception  and  approach  of  an  assessment  will  include  the  use  of 
hardware  that  is  appropriate  to  the  assessment  task,  but  this  alone  does  not  guarantee  that 
the  assessment  itself  will  be  simple  and  clear  in  concept  and  approach.  In  general, 
hardware  that  supports  management,  planning,  and  the  networking  of  administrators  and 
analysts  will  be  needed  to  provide  the  coordination  of  all  the  resources  involved  in  an 
elegant  assessment. 


Software  Promoting  Simplicity 

Software  means  for  encouraging  simplicity  and  clarity  in  the  design  and  execution  of 
assessments  also  focus  on  software  that  enhances  program  management  and 
communications.  Such  software  can  provide  the  links  between  administrator  and  analyst, 
analyst  and  other  analysts,  and  all  assessment  resources  necessary  for  the  coordination  of 
the  assessment  task.  It  can  also  enable  administrators  and  analysts  to  keep  track  of  the 
resources  being  used  in  an  assessment  and  provide  the  means  of  documenting  and 
evaluating  the  relationships  between  resources.  Of  course,  the  use  of  well-designed 
software  that  is  itself  simple,  clear,  and  matched  to  the  analysis  tasks  involved  are  simple 
in  concept  and  approach.  Some  of  the  characteristics  of  such  software  were  addressed 
when  the  goal  of  meeting  expectations  for  the  use  of  appropriate  resources  in  assessments 
were  considered. 
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Procedures,  Processes,  And  Methodologies  Promoting  Simplicity 

Supplying  an  assessment  that  is  simple  and  clear  in  conception  and  approach 
involves  the  use  of  procedures  that  support  the  availability  of  well-designed  and  proven 
hardware  and  software  and  well-trained  and  experienced  administrators  and  analysts.  But 
in  addition  to  the  rather  strict  application  of  procedures  usually  associated  with  model 
validation  and  verification  and  training,  the  achievement  of  simplicity  and  clarity 
involves  the  need  for  flexibility  in  procedures  and  the  freedom  to  improvise  and  innovate 
as  an  assessment  proceeds.  In  the  course  of  performing  an  assessment,  unexpected 
discoveries  or  the  availability  of  new  hardware  such  as  an  actual  threat  missile  system 
may  provide  a  means  of  greatly  simplifying  the  assessment.  Procedures  should  not 
inhibit  the  introduction  of  simplifying  ideas  and  resources,  but  neither  should  they  allow 
for  the  casual  abandonment  of  an  approach  until  the  alternative  is  proven  to  merit 
consideration.  The  need  to  make  judgments  regarding  the  merit  of  approaches,  data, 
tools,  and  other  resources  as  an  assessment  is  planned  and  executed  suggests  a 
requirement  for  procedures  that  allow  for  incremental  and  pilot  assessments  prior  to 
making  a  major,  full-scale  assessment  involving  the  commitment  of  substantial  resources. 


Personnel  Issues  Associated  With  Simplicity 

Supplying  assessments  that  are  simple  and  clear  in  concept  and  approach,  especially 
when  one  means  of  supplying  such  an  assessment  is  to  provide  for  flexibility  and 
innovation,  requires  trust  and  cooperation  between  the  planners  and  executors  of  the 
assessment.  Such  trust  is  encouraged  when  both  administrators  and  analysts  are 
confident  of  their  own  training  and  the  availability  of  resources  required  for  a  quality 
assessment.  It  is  also  encouraged  when  suppliers  understand  and  appreciate  the  needs  of 
customers  throughout  the  acquisition  hierarchy,  and  when  customers  understand  and 
appreciate  the  needs  of  assessment  suppliers  at  all  levels.  Gaining  such  understanding 
involves  the  thorough  and  ongoing  training  of  administrators  and  analysts  in  their 
respective  fields  and  the  cross-training  of  administrators  in  the  field  of  analysis  and 
analysts  in  the  field  of  administration.  Such  cross-training  could  be  accomplished  with 
exchange  tours  between  the  offices  of  the  administrators  and  analysts.  Finally,  the 
elegant  design  and  execution  of  assessments  calls  for  training  of  both  administrators  and 
analysts  in  the  scientific  method  and  design  of  experiments. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  simplicity  in  concept  and  approach  is  presented  in  Table  14. 

TABLE  14.  Means  (Resources)  for  Meeting  and  Exceeding  Expectations 
Associated  with  Simplicity  in  Concept  and  Approach. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  simplicity  in  concept  and  approach 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Self  contained 

User-friendly  models 

Minimize  number  of 

Administrators  and 

Self  sufficient 

Graphics 

suppliers 

analysts  trained  in 

Low  overhead 

Minimize  models  and 

Analysis  approach 

scientific  method  and 
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TABLE  14.  (Contd.) 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

model  interfaces 
Program  management 
software 

scaled  to  problem 

Flexible  analysis  approach 

Pilot  assessments 

Assessment  process 

Verified  and  validated  tools 

design  of  experiments 
Cross-training  between 
analysts 

Exchange  tours  between 
analysts  and 
administrators 

MINIMAL  ADMINISTRATIVE  BURDEN 


Issues 

Bureaucratic  interfaces  (AirTasks,  Memoranda  of  Understanding  (MOU),  inter- 
Departmental  tasking  agreements),  standards,  licensing  agreements,  security 
requirements,  travel,  and  contracting  requirements  are  elements  of  the  administrative 
burden  associated  with  performing  assessments  as  part  of  the  weapon  systems  acquisition 
process.  In  general,  administrative  tasks  do  not  contribute  directly  to  the  accomplishment 
of  assessments,  but  they  must  be  performed  to  acquire  and  maintain  hardware  and 
software,  to  implement  and  control  procedures,  and  to  pay  and  train  personnel. 


Hardware  To  Minimize  The  Administrative  Burden 

The  most  direct  approach  to  minimizing  the  administrative  burden  is  to  limit  the 
number  of  customers  and  suppliers  involved  in  an  assessment.  In  terms  of  hardware,  this 
suggests  limiting  the  types  and  quantities  of  computers  used  to  model  the  candidate 
system.  It  means  limiting  the  facilities  associated  with  the  assessment.  To  avoid  travel 
and  duplication  of  hardware  and  software,  it  may  mean  linking  facilities  and  hardware 
with  data  networks  or  the  development  and  use  of  teleconferencing  centers. 


Software  To  Minimize  The  Administrative  Burden. 

Software  issues  related  to  minimizing  the  administrative  burden  include  limiting  the 
types  and  quantities  of  models  used  in  the  assessment;  using  models  that  are  pre¬ 
approved  or  “accredited”  by  the  customer  (so  the  tasks  of  validating,  verifying  or 
justifying  the  use  of  models  during  the  assessment  will  be  eliminated);  using  models 
written  in  common  languages  that  can  be  compiled  and  run  on  a  wide  variety  of 
computers;  using  models  that  have  user-friendly  input/output  (I/O)  interfaces  that  will 
allow  for  the  direct  linkage  of  model  to  model,  or  model  to  hardware;  and  the  direct 
linkage  to  common  databases.  A  model  or  model  set  with  many  of  these  characteristics  is 
the  goal  of  the  model  integrating  architecture  (previously  mentioned  as  a  means  to  meet 
the  expectations  of  consistency  and  accuracy). 


Procedures,  Processes,  And  Methodologies  Minimizing  The  Administrative  Burden 

Procedures  that  seem  to  offer  promise  as  means  to  minimizing  the  administrative 
burden  are:  (1)  identifying  a  single  entry  point  for  initiating  assessments  in  the 
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assessment  supplier's  organization,  (2)  developing  a  “certificate”  program  for  analysts  so 
customers  can  be  assured  their  assessment  resources  will  not  be  consumed  in  the 
administrative  tasks  of  training  analysts  and  acquiring  hardware,  (3)  developing  standard 
forms  for  assessment  AirTasks  and  MOUs,  and  (4)  using  a  single  process  for  the  internal 
review  of  assessment  products.  Another  key  procedure  for  reducing  administrative  tasks 
is  the  requirement  that  all  software  be  supported  with  complete  and  up-to-date 
documentation.  Procedures  such  as  department-  or  Center-level  review  of  assessment 
tasks  and  products  should  also  be  considered  as  a  means  of  encouraging  coordination 
among  assessment  suppliers. 


Personnel  Issues  Associated  With  Minimizing  the  Administrative  Burden 

To  help  relieve  the  administrative  burden,  administrators  and  analysts  need  to  be 
trained  in  the  weapon  system  acquisition  process.  Administrators  need  to  appreciate  the 
unavoidable  and  necessary  administrative  requirements  involved  in  the  ownership  of 
complex  facilities,  secure  computers,  computer  software,  and  the  constant  need  to  train 
analysts  in  the  state-of-the-art  of  analysis  and  military  technology.  To  the  extent 
common  databases  and  networked  models  are  available,  analysts  need  to  be  proficient  in 
the  use  of  such  resources.  Analysts  and  administrators  need  to  respect  the  needs  for  a 
single-point-of-entry  to  assessment  suppliers,  or  at  least  control  the  way  customers 
interface  with  suppliers. 

A  summary  of  the  resources  associated  with  meeting  and  exceeding  the  expectation 
of  minimal  administrative  burden  is  presented  in  Table  15. 


TABLE  15.  Means  (Resources)  for  Meeting  and  Exceeding  Expectation 
of  Minimal  Administrative  Burden. 


Means  (resources  required)  to  meet  and  exceed  expectation  of  minimal  administrative  burden 


Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Minimize  types  and 
quantities  of  hardware 

Networked  facilities  and 
hardware 

Teleconferencing  center 

Single  model 

Certified  models 

Minimize  types  and 
quantities  of  models 

Model  integrating 
architecture 

Single  analysis  supplier 

Certified  suppliers 

Minimize  suppliers 

Control  of  entry  points 
to  analysis  process 

Standard  assessment 
AirTasks  and  MOUs 

Department/Center 
review  of  assessment 
tasks  and  products 

Analysts  trained  in 
weapon  systems 
acquisition 

Administrators  familiar 
with  capabilities  of 
suppliers 
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SUMMARY  OF  PROGRAM  MANAGER  AND  ANALYST  EXPECTATIONS 


This  assessment  of  expectations  has  been  undertaken  to  provide  managers  of  COEAs 
and  those  performing  the  analysis  involved  in  COEAs  a  guide  for  the  decisions  to  be 
made  when  defining  and  performing  a  COEA.  This  guide  supports  the  decision-making 
process  by  listing  ten  essential  expectations  program  managers  and  analysts  may  have  of 
a  quality  COEA.  These  expectations  are,  in  effect,  criteria  by  which  the  choices  made 
during  a  COEA,  (ranging  from  who  should  be  included  on  the  COEA  “team”  to  which 
models  and  data  should  be  used),  may  be  justified.  Applying  these  criteria  effectively 
involves  asking  at  each  decision  point  whether  or  not  the  resulting  activity  will  support  a 
COEA  that  is: 

1.  Consistent  (Repeatable,  Methodological  Integrity) 

2.  Accurate  (Accredited  Models  and  Data;  Trained  and  Experienced 
Analysts  and  Administrators) 

3.  Robust,  (comprehensive  results) 

4.  Well  documented 

5.  Affordable 

6.  Timely 

7.  Accessible 

8.  Appropriate  in  the  use  of  analysis  resources 

9.  Simple  in  concept  and  approach  (understandable) 

10.  Produced  with  minimal  administrative  burden. 

These  ten  expectations  are  distilled  from  several  lists  of  expectations  and 
measurement  criteria,  and  the  distillation  process  is  presented  in  the  body  of  this 
document.  Whether  or  not  there  should  be  ten  or  twenty  essential  expectations,  and 
whether  or  not  the  ten  identified  do,  in  fact,  represent  the  essential  expectations  is  of 
course  open  to  criticism.  In  defense  of  the  ten  selected,  they  were  developed  using  first 
principles  presented  in  this  report. 

The  point  is,  it  is  essential  that  program  managers  have  confidence  in  the  COEA  for 
which  they  are  responsible.  They  need  to  know  it  was  performed  to  high  analytical 
standards  and  the  conclusions  presented  can  be  defended  clearly  and  honestly.  They  need 
to  know  the  resources  they’ve  spent  on  the  COEA  were  necessary:  that  they  were 
expended  in  a  prudent  and  reasonable  way.  If  the  ten  expectations  addressed  above  are 
met,  program  managers  can  be  confident  they  will  have  this  knowledge. 

Both  analysts  and  program  managers  need  to  have  a  clear  justification  for  the  effort 
and  resources  they  expend  in  performing  a  COEA.  Analysts  need  to  know  what  is 
expected  of  them  by  the  COEA  customer,  and  they  need  to  be  able  to  defend  their 
expectations  of  what  they  consider  quality  analysis.  They  need  to  be  able  to  negotiate 
with  the  customer  over  which  expectations  take  priority  as  the  COEA  is  performed.  In 
particular,  program  managers  and  analysts  need  to  be  able  to  work  with  each  other  to 
resolve  the  inevitable  tension  arising  between  the  customer’s  expectations  for  timely, 
inexpensive  analysis  and  the  analyst’s  professional  stake  in  accurate,  robust  analysis. 
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There  can  be  disagreement  regarding  priority  of  all  the  expectations,  and  their 
priority  is  likely  to  change  from  COEA  to  COEA.  Hopefully,  this  guide  defines  the  ten 
expectations  completely  enough  so  program  managers  and  analysts  have  a  sound  base  on 
which  priority  may  be  negotiated. 

A  summary  of  the  expectations  of  assessment  customers  (program  managers)  and 
suppliers  (analysts)  and  the  means  (resources)  required  to  meet  those  expectations  is 
presented  in  the  Appendix. 
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APPENDIX 


TABLE  A-l.  Expectations  of  Assessment  Customers  and  Means  (Resources)  for 
Meeting  and  Exceeding  the  Expectations. 


Means  (resources  required)  to  meet  and  exceed  expectation 


Expectation 

Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Consistent 

Limit  types  and 
quantities  of  hard¬ 
ware  used 

Minimize  types  of 
computers  and 
operating  systems 

Same  model 

Same  model  version 

Same  data 

Minimize  models  and 
databases 

Configuration  manage¬ 
ment  software 

Model  integrating 
architecture 

Configuration 
management  in 
accordance  with 
military  standards 
Department/Center 
review  of  assessment 
tasks  and  products 
Apply  documentation 
standards  to  models, 
databases,  and 
analysis  products 
Assessment  process 

Analysts  trained  in 
use  of:  models, 
databases,  and 
computers;  local 
analysis  process; 
fundamentals  of 
weapon  system 
acquisition 
Administrators 
trained  in  weapon 
system  acquisition 
process;  funda¬ 
mentals  of 
analysis 

Training  in  the 
scientific  method 
and  design  of 
experiments 

Accurate 

High  capacity 

High  speed 
Hardware-in-the-loop 
Operator-in-the-loop 
Special  facilities 
Networked  hardware, 
operators,  facilities 
Exploited  hardware 

Emulation  quality 
software 

Minimize  data  handling 
Software  to  link 
hardware,  facilities, 
and  operators 

Model  integrating 
architecture 

W&A 

Standard  test  cases 
Analysts  perform 
quality  checks 
Analysis  reviewed  by 
technical  experts 

Analysts  trained  in 
specific  types  and 
components  of 
weapon  systems 
Training  in 
performing 
integrated 
assessments 
Training  in  the  use  of 
all  analytical  tools 

Robust 

Hardware-in-the-loop 
Operator-in-the-loop 
Exploited  hardware 
Range  tests 

Field/fleet  exercises 

Networked  hardware 

"Mission"  analysis 
capability 

Model  full  spectrum  of 
natural  and  man¬ 
made  environment 
Model  C3I  links 
Interactive 

W&A 

Standard  scenarios 
Customer/ supplier 
reviews 

Coordination  between 
assessment  suppliers 
Assessment  process 

Administrators  and 
analysts  trained  in 
military  operations 
Participation  in  field 
exercises 

Analysts  cross- 
trained  in  weapon 
system  types  and 
tactics 
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TABLE  A-l.  (Contd.) 


Means  (resources  required)  to  meet  and  exceed  expectation 


Expectation 

Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Documented 

Personal  computers 
Workstations  with 
capacity  for  word 
processing  and 
graphics 

Word  processing, 
spread  sheet,  and 
presentations 
software 

Word-processing 
software  tailored  to 

TP  or  TM  format 
Graphics  outputs 
Statistics  post¬ 
processors 

Document  review 
Performance  plans 
requiring  documen¬ 
tation  of  analysis  to 
meet  expectations 
Standard  documen¬ 
tation  format 

Level  of  documentation 
linked  to  level  of 
analysis 

Documented  databases 
Documented  software 
Assessment  process 
that  includes 
documentation 

Analysts  trained  in 
statistics,  docu¬ 
mentation,  design 
of  experiments, 
and  local  and 
headquarters 
analysis  review 
processes 

Administrators 
trained  in  docu¬ 
mentation  review 
and  performance 
planning 

Affordable 

Low  overhead 

Fast 

Networked  hardware 

Reliable 

Short  repair  and/or 
maintenance  time 
Supports  multiple  users 

Low  overhead 

Fast 

Flexible 

Program  management 
software 

On-line,  up-to-date 
management 
information  system 

Budget  presentation 
and  analysis  software 

Minimize  computer 
usage 

Optimize  hardware-in- 
the-loop  and/or 
interactive 
requirements 
Minimize  number  of 
suppliers 
‘Tag  along” 

arrangements  with 
field  tests 

Administrators 
trained  in  the  cost 
of  owning  and 
operating 

simulation  and  test 
capabilities 

Analysts  trained  in 
the  proficient  use 
and  maintenance 
of  their  models 
Administrators  and 
analysts  certified 
in  their  disciplines 

Timely 

Fast  (parallel,  optical) 
User  friendly 

Flexible 

Networked  hardware 
Short  repair  and/or 
maintenance  time 
Supports  multiple  users 

Fast 

Optimum  architecture 
Model  integrating 
architecture 

Program  management 
software 

On-line,  up-to-date 
management 
information  system 

Minimize  number  of 
suppliers 

Minimize  model  runs 
Minimize  reviews 
Provide  quality  control 
at  analysis  level 
Coordinated 

customer/supplier 

interfaces 

Certified  analysts 

Administrators 
proficient  in 
weapon  systems 
acquisition 

Analysts  proficient 
in  analysis  process 

Accessible 

High-volume,  high¬ 
speed  storage  and 
retrieval 

Secure 

Networked  hardware 

User  friendly 

Configuration 

management 

Models  with  post¬ 
processors  for  library 
functions 

Well  defined  I/O 
interfaces 

File  server 

World  Wide  Web 
access  to  models, 
data,  briefings, 
program  definition 

Access  to  data  libraries 
controlled  and 
monitored 

Single  library 
administration 
Standard  file  naming 
convention 

Resources  dedicated  to 
library  development 
and  maintenance 

Administrators 
trained  in  use  of 
on-line  libraries 
Analysts  trained  in 
library  develop¬ 
ment  and 
maintenance 
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TABLE  A-l.  (Contd.) 


Means  (resources  required)  to  meet  and  exceed  expectation 


Expectation 

Hardware 

Software 

Procedure,  process, 
methodology 

Personnel 

Appropriate 

analysis 

resources 

Hardware  suited  to 
software 

Hardware  use 
proportional  to 
analysis  problem 

Hardware,  facilities, 
exercises  sensitive  to 
environmental 
constraints 

Networked  hardware 

State-of-the-art 

software 

Coordination  of  model 
and  data  fidelity 
Model  integrating 
architecture 

Program  management 
software 

Catalogue  of  models 
Catalogue  of  facilities 
Calendar  of  exercises, 
tests,  TECHEVALs, 
OPEVALs, 

Dedicated  resources  for 
software 

development,  testing, 
maintenance, 
validation,  and 
verification 

Software 

documentation 

Coordination  of 
software 

development  through 
national, 

multiservice  bodies 
(e.g.,  JTCG/AS, 
JTCG/ME,  EMCC, 
IACs,  JCG-EW 

Administrators  and 
analysts  trained  in 
scientific  method 
and  design  of 
experiments 
Administrators  and 
analysts  trained  in 
full  spectrum  of 
analysts  resources 
Analysts  trained  in 
the  weapon  system 
acquisition 
process  and 
familiar  with  the 
questions  asked  at 
program  milestone 
reviews 

Headquarters  -  to  -field 
exchange  tours 
Exchange  tours  for 
analysts 

Simple  in 
concept  and 
approach 

Self  contained 

Self  sufficient 

Low  overhead 

User  friendly 

Graphics 

Minimize  models  and 
model  interfaces 
Program  management 
software 

Minimize  number  of 
suppliers 

Analysis  approach 
scaled  to  problem 
Flexible  analysis 
approach 

Pilot  assessments 
Assessment  process 
Verified  and  validated 
tools 

Administrators  and 
analysts  trained  in 
scientific  method 
and  design  of 
experiments 

Cross  training 
between  analysts 
Exchange  tours 
between  analysts 
and  administrators 

Minimum  ad¬ 
ministrative 
burden 

Minimize  types  and 
quantities  of 
hardware 

Networked  facilities 
and  hardware 
Teleconferencing 
center 

Single  model 

Certified  models 
Minimize  types  and 
quantities  of  models 
Model  integrating 
architecture 

Single  analysis  supplier 
Certified  suppliers 
Minimize  suppliers 
Control  of  entry  points 
to  analysis  process 
Standard  assessment 
AirTasks  and  MOUs 
Department/Center 
review  of  assessment 
tasks  and  products 

Analysts  trained  in 
weapon  systems 
acquisition 
Administrators 
familiar  with 
capabilities  of 
suppliers 
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ACRONYMS 


C^I 

command,  control,  communications,  and  intelligence 

CAD 

computer-aided  design 

CNO 

Chief  of  Naval  Operations 

COEA 

Cost  and  Operational  Effectiveness  Analysis 

DOD 

Department  of  Defense 

EMCC 

Electromagnetic  Code  Consortium 

I/O 

input/output 

IACs 

Information  Analysis  Centers 

JCG-EW 

Joint  Coordinating  Group  for  Electronic  Warfare 

JTCG/ME 

Joint  Technical  Coordinating  Group  for  Munitions 
Effectiveness 

LFT&E 

live  fire  test  and  evaluation 

MOU 

Memoranda  of  Understanding 

M/S 

modeling  and  simulation 

NAWCWPNS 

Naval  Air  Warfare  Center  Weapons  Division 

OPEVAL 

operational  test  and  evaluation 

OPNAV 

Office  of  the  Chief  of  Naval  Operations 

OT&E 

operational  test  and  evaluation 

PEO 

program  executive  officers 

PMA 

Program  Manager  (in  NAVAIR) 

SECDEF 

Secretary  of  Defense 
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